Practical Research IDEs 2020PRIDE 2020

Numerous program analyses have been presented at research conferences over the years, typically with detailed analysis using appropriate specific metrics to show that the new analysis improves results compared to previous approaches. While such evaluation shows the new analysis improves analysis per se, it is not always entirely clear how much impact these improvements would have on an overall system of which the analysis is one component. Our focus, in particular, is on one especially challenging aspect of this: programming tools and understanding how much new analysis improves the information displayed by a tool. In this case, creating the needed tools is a challenge since it requires mastering the workflows and idioms of IDE tools and integrating the analysis code into the IDE. While most IDE tools have plugin architectures to make this process more manageable, even plugins often require significant expertise and programming. Worse yet, there is much diversity in IDE tools, and nothing that can be considered dominant except in certain niches, e.g. Xcode for iOS and AndroidStudio for Android. On the one hand, it does not seem reasonable to expect analysis experts to expend the effort to master and use diverse IDE ecosystems; on the other hand, it would be nice to be able to evaluate new analysis work in the context of IDE usage.

The recent MagpieBridge system provides a potential way out of this dilema; MagpieBridge provides a way to connect arbitrary program analyses to most popular IDE tools, in particular any that support the Language Server Protocol. Experiments with MagpieBridge (published at ECOOP 2019) have shown its ability to integrate several existing analyses into many IDE tools with minimal effort. These analyses include FlowDroid, CogniCrypt and Ariadne, and similar analyses. These analyses span multiple languages—Java, JavaScript, Python—multiple domains—privacy, cryptography, data science—multiple analysis frameworks—WALA and Soot—and have been shown in many IDE tools—Eclipse, Visual Studio Code, IntelliJ—and editors—Sublime Text, Microsoft Monaco. The analyses have been integrated into all these tools using MagpieBridge with glue code comprising a couple of hundred lines of code in total that provides rich information such as highlighting program traces and enabling quick fixes. Thus, MagpieBridge could allow numerous analyses to easily render their results in many IDE tools to provide a new form of evaluation. If such evaluation became standard where appropriate, it could greatly accelerate the process of incorporating novel analyses into existing tools.

The purpose of this workshop would be to assess how amenable work appearing at recent conferences is to being incorporated into modern IDE tools, and to actually incorporate at least some of the work to the extent possible. To facilitate this process, we propose a hack-a-thon style workshop in which the creators of novel analyses would work with the creators of MagpieBridge to create IDE integrations of their results and show it in a range of IDE tools. We hope that the ease of IDE integration that MagpieBridge has enabled in out work so far will encourage others to assess, as part of their analysis work, how easily it can also be included in an IDE. To provide a demonstration, we propose to invite the authors of relevant papers to submit their accepted papers to PRIDE, and work with the MagpieBridge authors at PRIDE to integrate their work into IDE tools. To highlight any successful IDE integration of papers, we propose that such integrations be shown at the poster session.

Call for Participation

Goal

The goal of this workshop is to help researchers take their research developments and integrate them easily into existing IDE tools. We are seeking participation from researchers who have developed novel analysis technology and would like an easy way to deploy this technology in existing IDE tools.

For example, a new, more-precise taint analysis is nice, but it will be potentially much easier to use and to evaluate if it is available in popular IDE’s—e.g. IntelliJ, VisualStudio Code, PyCharm, JupyterLab—in editors—e.g. Subime Text, Atom—and traditional editors used by programmers—e.g. Emacs, Vim.

MagpieBridge provides support for accomplishing this: it provides a framework in which a research analysis can be invoked from standard IDE’s. The underlying mechanism makes use of the Language Server Protocol (LSP), but that is completely hidden by a straightforward API that allows analyses to be invoked on source projects from IDE’s and report results that get displayed in idioms natural to each IDE.

Such functionality could benefit a wide range of research tools, and so we encourage participation by bringing either your own research analysis or any analysis that you would like to see in a tool. To participate, it suffices to indicate your interest by email to the organizers, including a description of the analysis you would like to integrate into tools.

MagpieBridge is designed to be broadly useful, but there are a few guidelines as to what sort of analysis would likely work best. Consult these guidelines to understand whether your analysis would be easy to integrate and whether you could display information as you would like.

Performance and language requirements.

• MagpieBridge is largely asynchronous with respect to the IDE, so analysis computation time will never block the IDE and results are reported when they are ready. Latency-sensitive features such as hovers can have results computed during analysis, so analysis computation time can be hidden even there. This increases the amount of analysis time that is tolerable compared to on-demand analysis; however, analyses that take excessive amounts of time would not be suitable.
• MagpieBridge is written in Java, which makes integrating analyses also written in Java simplest. However, the protocol for communicating with analyses is straightforward, and we will support projects communicating with other languages with JNI bindings when appropriate and also with JSON messaging when that is most convenient. As such, the language in which your analysis is written should not be an issue for taking part in the hackathon.

How analysis results can be displayed.

• MagpieBridge largely supports functionality provided by LSP, with a focus on three features given below, but also provides a mechanism to integrate arbitrary HTML including forms and JavaScript. The three focus features are the following:
  • Diagnostics are the kind of error, warning and informational messages commonly supported in IDE's. MagpieBridge provides an API for returning these kinds of messages, which then get displayed by the IDE using its usual mechanisms.
  • Hovers are messages that pop up when a user lets the mouse hover in a specific location. Note that LSP in most IDE's integrates its hovers seamlessly with any other hover messages that the IDE generates.
  • Code lenses are comments injected into program buffers by the IDE, typically distinguished by using a different font or color. These lenses can be linked to actions that interact with the analysis.
• MagpieBridge does not support wholesale changes to the IDE, but, using the HTML integration mechanism, it allows non-trivial views to e.g. configure an analysis or display results. This HTML integration is robust across all IDE's supported: if an IDE provides such a mechanism, then MagpieBridge uses it, and otherwise these views show up in the user's Web browser.

How analysis results can be used.

• MagpieBridge supports the LSP notion of code fixes, in which an analysis provides a code range from the program and the text with which it would like to replace that range. This is the primary mechanism, but it is straightforward to extend it to multiple ranges for a single fix.
• MagpieBridge supports arbitrary HTML documents, which can be used by an analysis to display more complex information than can be easily rendered in a hover. For example, a detailed description of a security warning could be display for a brief error message.