dpi1-v1.0 (draft 9, with implementation) by: Jorge Arellano, Eric Gaudet Jan 31, 2002 /* This document is an hybrid between a draft and a doc. This is because the full dpi1 spec isn't completely finished yet, but we already have a working framework to develop with. */ --------------------- Simple plugins (dpi1) --------------------- ------------ Introduction ------------ Hopefully, dillo will have two types of plugins: one of a simple nature (described in this document), that operates in a similar way of CGIs (with some extensions), and another that integrates interaction with the rendering engine and its widgets. The fact is that our current rendering system, and its associated internal layers is still under construction, makes almost impossible to begin working on dpi2, and on the other hand, gives an excellent oportunity to focus on dpi1 (that's also expected to be easier to program with). We expect dpi1 to be incrementally extensible, but also very simple in its concept at the same time. Please bear the simplicity concept in mind while studying this document. That's the basis for a powerful and clean dpi spec. ---------- Motivation ---------- Dillo aims to be a small, fast and efficient web browser. This kind of PI is designed to push out of the main code features that usually are optional, and also some others that are required (as bookmarks for instance). The design idea is to keep the main code-base small and to simplify the development of new features, by not requiring a PI programer to know the whole internals of dillo. This way, extensions can be easily coded, and there's also place for developing unrelated programs that can use dillo as a GUI! -------- Overview -------- Technically, the plugin program is a coprocess (an independent program spawned by the browser) that communicates with the browser through e certain channel. The current implementation uses unix domain sockets: --------- -------------> ------ | Browser | | socket | | PI | --------- <------------- ------ (Note: a Unix domain socket may be used as well) The main idea is to have a bidirectional channel on which the browser and the plugin can communicate. The data flow is encapsulated in a small but flexible protocol (dpi protocol) that resembles HTML tags. For instance, for sending an status message from the plugin to the browser: --------------------------- Note that with this scheme: --------------------------- * PI-programs can be one-demand/one-response, and there's space for multiple negotiations between the browser and the plugin too. * Status/progress messages can be passed back to the browser. * Browser and plugin can request information from each other, and they can also set data! * It is possible to handle persistent dpi1 processes, though not indispensable. * It's easy to extend the protocol by adding new operations without affecting backward compatibility. * There's plenty of choice for PI languages. ----------------- How does it work? ----------------- In general terms, the browser starts (or contacts) the PI when it finds something that belongs to it (as an URI), and handles it to the PI. The PI receives the request, processes it and answers back in the form of an HTML page, or status message, or a dpi command. A simple session (one-demand/one-response) can look like this: 1.- The browser starts the PI when it finds an URI that belongs to the PI. 2.- The PI starts. 3.- The browser handles the URI to the PI with a dpi command. 4.- The PI composes its answer (an HTML page) and sends it back within dpi protocol datagrams. 5.- The PI acknowledges the browser when done. ------------- Some examples ------------- Before getting into the protocol itself, it may be ilustrative to look at some functionality that could be handled using this kind of PIs. For those willing to participate in the protocol developing process, it's certainly adviced to keep these examples in mind for testing design ideas. 1) Downloads * current cache allocates them in memory, so handling them to an external program (PI) has several advantages. * the PI chooses which agent (wget, curl, ...) to use, and sends feedback to dillo's download interface (HTML). * Downloads don't stop if dillo crashes. 2) FTP * to browse FTP directories (PI outputs HTML). * Downloads using 1). * progress messages (as loggin in, etc) can go to the status bar. 3) Bookmarks * The BM PI maintains the BM database (a single file!). * The PI builds an HTML page for dillo. * Needs coomunication with the browser (URI, titles, add bookmark, etc). * Some of the functions can be handled with FORMS. * Browser may clean BM-pages out of the stack, for them not to show in history. * Exporting BM becomes a matter of Save-page-as! NOTE: All of this has been already implemented (BM). 4) Preferences (AKA configuration) * PI requests information from the browser (current prefs). * PI commands the browser to change its "state" * PI maintains dillorc * As 3), may require stack-cleanning of page. 5) Info reader. * Just a handy info to html filter (very simple). * one-demand/one-response. 6) Doc viewer (?) * uses an external program to convert doc to html 7) Help browser * A PI that knows of local documentation, presents an index and probably a search feature! 8) A handler PI for "mailto:". (...) Well, that's the idea of simple PI. A way of extending dillo to suit end-user needs, without requiring extensive knowledge of its internals. Note also that these extensions don't make into the core of the browser. ------------------------- Developing a dillo plugin ------------------------- I think the best way to get started will be to explain the basics of "dillo <=> PI" communication, then review the already implemented tags of the dpi protocol, give some tips, and finally let you examine the code of the bookmarks plugin (a full application that's highly commented). ----------------- Dpi communication ----------------- ------------ dillo => PI: ------------ There're three ways: 1.- dpi tag 2.- static URL 3.- dynamic URL (with form data) Note that all of them are in fact dpi tags, the division is made to separate some concepts. 1.- The first one is for any specific dpi command sent from the browser to a PI. For example: 2.- The second one usually refers an URL "inside" the PI's scope and commands it to do something (usually to send back a page). For intance: In this case the PI must react sending the main page for the user's bookmarks. 3.- The third one is the most versatile. It sends back (to the PI) all the data associated with a FORM that was most usually sent by the PI itself. For instance: (all in a single line) That's how the PI know we want to modify the first section's name of the bookmarks. ------------ PI => dillo: ------------ There're two ways: 1.- dpi tag 2.- HTML page 1.- The first one is the generic mechanism and usually serves to command the browser. For instance: 2.- The second one tells the browser we'll send an HTML page. ------------ The protocol ------------ Amazingly (if you've already tried the bookmarks PI), all its functionallity is carried on with these few dpi tags: dillo => PI: PI => dillo: So it is NOT that hard! ---------------------------- Tips for developing a new PI ---------------------------- First, examine the above mentioned protocol tags, then read the code of hello_world PI and figure out its work. After that, it'll be clear what the basic skeleton is, and how to send back a page to dillo. The next step is to make a FORM, get some data from the user, and to send back a new page that manipulates the input data. For this: 1.- Design an HTML page with the form 2.- Design an HTML page with the desired answer skeleton 3.- Make your PI send it as the main page (as an answer to "dpi:/mydpi/", for instance) Once you have that, you'll need to process the input data: 4.- Make your PI print what it receives so it becomes clear what you're dealing with. 5.- Either try to process it right away, or study how it's done in the 'Bmsrv_parse_buf' function of the bookmarks PI. Now you have the parsed data: 5.- Manipulate it somehow 6.- Build the answer page and send it back If you made this far, the rest will be clear enough from studying the commented bookmarks' code! ------------------------------------- What's missing in this implementation ------------------------------------- 1.- Some dpi commands 2.- PI start/initialization. 3.- All the menu handling (from PI to browser) is to be defined (there're some ideas about these near the end) 1.- Dpi commands ---------------- From the set of uninplemented dpi commands, I want to comment two: send_html_page: This was meant to encapsulate HTML chunks of certain length. The final chunk should be marked with length='0' or something akin. The main purpose of it is to allow communication after the HTML page is sent. That way, for instance, the bookmarks plugin could send a staus message adding more information about a performed operation (as "Deleted 3 bookmarks"). The only reason I didn't code it from the start was to get to a working framework as fast a s possible. It's not hard though. Currently the dpi framework uses 'start_send_page' wich assumes that all the following data, up to EOF, is part of the HTML page. send_data: This will be a generic way of exchanging information. Useful for sending/setting preferences for instance. Very similar to the previous command, and much like the current 'chat' command. BTW, the 'chat' command is not necessary (just gave a look to the funny and irrelevant talk between dillo and the bm PI!). It was just a way of testing bidirectional data exchange. If you need to send/request some data, feel free to use 'chat' until 'send_data' is implemented. 2.- PI start/initialization --------------------------- There's a lot more info about it in the following pages. A polished interface for registering/launching plugins is yet to be developed. Currently dillo launches the bookmarks plugin automatically, and no more. As said before this will change in the near future. 3.- Menu handling ----------------- Just read the following section!. ---------- ------------------------ SECTION II (this is the draft part) ------------------------ This section is mainly a gathering of ideas about what's not yet implemented: 1.- Dpi commands 2.- Menus 3.- PI initialization 4.- Plugins options 5.- Plan 6.- Future extensions 7.- Final notes ------------ Dpi commands ------------ dillo -> PI ----------- CMD : DpiAbort :: Sent by dillo to abort an "in-progress" plugin. Dillo might send a "kill -9" command to the PI if it doesn't answer. CMD : DpiBye :: The plugin is commanded to quit (after it's done). It may be used to end a "resident" PI that's doing nothing. Plugins can end and exit when finished or keep sleeping, but they MUST acknowledge dillo when they're done on a per client basis. CMD : DpiError :: Sent by dillo when a command is not understood. PI -> dillo ----------- CMD : DpiRequestInfo :: Sent to the browser to request information. For instance: - Dillo version - title of current page - URL of current page - preferences (the whole preferences as text) - All active colors (background, text, link, visited link) CMD : DpiSetData :: Sent to the browser to change its "state". Example: preferences, menus. CMD : DpiDone :: Sent by the PI when finished servicing a client. ----- Menus ----- Menus need to be defined thinking on the needs of current and future PIs. The may not require menu entries (as the FTP PI), may require a single menu entry, or a whole menu, or a button (maybe on the toolbar but smaller in size than the main ones). The key point here is to provide simple support for menus, and not to complicate the protocol with seldom required features. << simplified menu registration protocol --EG Main goals: - a plugin may or may not want to be registered in the menus when initializing: menus are optionnal. - a plugin may want to change some or all its menus entries at runtime. (example: adding/deleting/modifying a bookmark entry) - the plugins menu looks like this: there's a "Plugins" menu in dillo's menubar, where all plugins are shown, provided they asked for it during the init stage. Then it's up to the plugin to ask this entry to be an url or a submenu. Commands: CMD : DpiMenu Menu commands: - insert: add an entry named just after the entry ID. Dillo must answer with the new entry ID. - delete: delete an entry ID and all dependent sub-menus. - modify: modity the of an entry. - submenu: make an entry a submenu. - url: give as the url to be called when the menu entry is clicked. >> ----------------- PI initialization ----------------- There main problem with running each PI everytime dillo starts is the added startup time. If, for instance, there's a PERL and a Python PI, the time increase is too much. This scheme avoids running each plugin when dillo starts: * Having a 'pluginsrc' file with the initialization data of all the already registered plugins (rc file in ~/.dillo dir). * Having a "Register PI" option (in File menu probably) that opens a "select file" dialog, checks the selection to be executable, starts it, and sends a DpiInit datagram. * To avoid registering of any binary as a dillo-PI, a small challenge is included in the DpiInit datagram. * PI must answer something like the following information inside a DpiInitAnswer datagram: Name = bookmarks // mandatory prefix = bm:/ \ One of these postfix = / is mandatory SmallButton = BookMarks \ Optional SmallButtonAction = bm:/show / camps MenuEntry = View Bookmarks \ Optional MenuEntryAction = bm:/show / camps * After that's done, dillo must save that info to pluginsrc. With this command added to the protocol: dillo -> dpi: CMD : DpiInit :: It should include some challenge data to assert it's a dpi. and from dpi->dillo: CMD : DpiInitAnswer :: Data is composed of a list of name=value pairs separated with ';' just like in an URL (it must answer the challenge). Example: Name="bookmarks";prefix="bm";MenuEntry="View Bookmarks" --------------- Plugins options --------------- There're two different things here, the PI registration data (handled by dillo in the 'pluginsrc' file for every PI), and the specific options a specific PI-program may provide for tunning its behaviour. The later one should be managed by the PI program itself. ---- Plan ---- The old plan said: > Make a basic implementation (DpiUri, DpiSendData, DpiDone), and send a "hello world" HTML page. This is to assert that the whole theory discussed here works OK. Once that's done, begin implementing simple plugins (as one that returns a nicely formatted bookmarks page), and finally extend the implementation to every command described here. When that's done, it will be time to try to implement the preferences, bookmarks and FTP plugins using this scheme, and to fix/extend anything that's necessary. > Of it, near 70% is done! Now the plan is: Implement other plugins and extend/polish the dpi framework as its needed. When it has proven stable: make the 0.7.0 release! Note: dillo must be able to launch its plugins before a release is done. ----------------- Future extensions (needs a lot more thought) ----------------- 1.- Postfixed call to a Dillo-plugin Dillo plugins registered as such will be called if the loaded page's in "." matches a Dillo-plugin registration, or if the PI has registered the same MIME type of that content, and thus the PI acts as a file-processing filter. Dillo will call the Dillo-plugin sending to it the whole loaded file, and expecting some data back. This can be used as a file-to-html filter. Example: "http://my.server.org/mydirectory/myfile.rtf" will render the rtf file as html if a Dillo-plugin has registered itself as dealing with "rtf" files. ----------- Final notes ----------- The draft part of this material is not "written on stone", but provided as a structural basis on which to develop the simple PI protocol. Please take it with a pinch of salt, in the best spirit of a RFC. As you can see, there's plenty of material here, and discussing threads can get very large if posts are not carefully thought of. Please take your time to study and fully understand the material, and only submit well backed/tested ideas. Thanks a lot to all of those that read this far. :-)