tag:blogger.com,1999:blog-74959939052593383932024-02-08T12:15:17.911+01:00devel@roadsend.phpdevelopment of roadsend phpShannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-7495993905259338393.post-9589515604874456202010-08-09T16:31:00.000+02:002010-08-09T16:31:07.729+02:00Current StatusHey folks, since several people have spoken up about the lack of movement on the project, I thought I'd give a quick update.<br />
<br />
We've got plenty of people who are interested in seeing this project move forward - unfortunately, few willing or able to help with the actual development. I had hoped to pick up at least one or two other developers by now. This project is sufficiently audacious that I can't hope to achieve the goals I set for it by myself, in the time I have available for it. I'm happy with the work that's been done so far and I'd like to see it continue. If/when other developers start contributing, I'll be happy to get back to work on it. Until then, my contributions to it will be spotty.<br />
<br />
So, if you know some C++ and you'd like to see this forward, let me know!<br />
<br />
If you're interested in language development and you're willing to try something new, you can also check out another project I'm involved with, Crack: <a href="http://code.google.com/p/crack-language/">http://code.google.com/p/crack-language/</a>Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com10tag:blogger.com,1999:blog-7495993905259338393.post-53134008545988029982010-02-10T21:37:00.000+01:002010-02-10T21:37:51.274+01:00ReproductionI made a recent mailing list post with some thoughts and status updates, which I thought I'd reproduce here.<br />
<blockquote>The big wave in the small world of alternative PHP implementations is the announcement of HipHop from Facebook. I haven't seen the source code as of yet, so rather than make any comments on it, I'll just note that Facebook (for better or worse) didn't attempt any sort of cooperation with either Roadsend or phc (or any others, as far as I know) when starting their project, and highlight a humorous (but true) comment Paul Biggar (author of phc) mentions in a recent blog post [1]:<br />
<br />
"... I’m also slightly annoyed that people all of a sudden care about PHP compilers. I worked on one for 4 years and I could not convince anyone to give a shit. But now that its got the Facebook logo on it, all of a sudden PHP compilers are the greatest thing ever. Bah."<br />
<br />
Anyway, when we see the source, we can determine how closely the goals of HipHop coincide with those of Raven. I'm intrigued that they (apparently) opted for their own C++ runtime (as we did with rphp).<br />
<br />
For now though, here's what we've been up to and where we're heading. <br />
<br />
Recent work is all about our frontend. We've been working towards two goals: a fast, memory efficient parser, and generating a parse tree (with matching transformation API) mostly compatible with the one used by phc. Regarding the former, we've adopted an AST design that's very similar to the one used in Clang (using LLVM's memory pool, string pool, child node iteration technique, etc). Regarding the latter, this is because we plan to use the PHP source level analyzer and optimizer that is part of the phc project. While fleshing these changes out, a new tool for static analysis has been produced (rphp-analyzer) which parses and analyzes PHP source files, able to run the available passes, produce messages, and dump to XML.<br />
<br />
We're now to the point where the parser is rather PHP 5.2 complete, and we've already begun porting phc lowering passes to rphp. Unfortunately, LLVM IR code generation had been suspended while this work was going on. I think we're almost to the point where simple code generation can resume.<br />
<br />
So now that a lot of ground work has been completed, we're hoping that the real guts of the phc analyzer and optimizer can be ported into rphp. I say "ported", because even though both projects are in C++, for purposes of speed and efficiency[2] we've made several design decisions that make our data structures and API a bit different from those in phc. This means there is some work involved in moving e.g. a pass from phc to rphp, but on the whole we've tried to minimize this overhead and it seems to be working out well so far.<br />
<br />
Otherwise, we're setting up a testing framework for the passes, and the grammar is still waiting for PHP 5.3/6 language constructs and general improvements. Another important goal I'd like to get to soon is making sure we can produce good diagnostics and error messages during parse and transform.<br />
<br />
Because we've been focused on the frontend, the runtime hasn't changed recently. Now that we're almost ready to resume code generation, though, we'll get back into that as well.<br />
<br />
In a nutshell, that's where we're at, and what we'll be working on for the time being. <br />
<br />
Finally, if you've been following Roadsend PHP at all and are dismayed at the speed with which we're progressing, it's worth noting that since this project went open source in 2007, it has had no financial support and is simply a grassroots labor of love[3] for those who contribute. If you're interested in lending a hand, join us in #roadsend on irc.freenode.net!</blockquote><blockquote>[1] <a href="http://blog.paulbiggar.com/archive/a-rant-about-php-compilers-in-general-and-hiphop-in-particular/">http://blog.paulbiggar.com/archive/a-rant-about-php-compilers-in-general-and-hiphop-in-particular/</a><br />
[2] Ostensibly, anyway .. we don't have real benchmarks yet<br />
[3] Ok, "enjoyable distraction", at least :) </blockquote>Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com11tag:blogger.com,1999:blog-7495993905259338393.post-63864784359808285422010-01-26T19:38:00.003+01:002010-01-26T19:44:55.464+01:00Front EndedQuick update:<br /><br />Between 12/14 and 1/14 we managed 90 commits worth of parser and other front end improvements.<br /><br />rphp-analyzer is now created during the build. This is a new tool just for static analysis, which we're using to help perfect the parser. We can now parse quite a bit of standard PHP5 code, and we've started to port several passes from the phc project into the rphp framework.<br /><br />We expect basic code generation to find its way back into the build soon!Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com4tag:blogger.com,1999:blog-7495993905259338393.post-17072108089971805582009-12-07T15:48:00.003+01:002009-12-07T17:07:13.482+01:00DataflowAttention has turned back to the compiler frontend. Some recent work includes:<br /><ul><li>Removal of the preprocessor. This was used (pre-parse) to convert double quoted strings into their single quoted equivalents, interpreting escape codes and interpolated variables in the process. This has been removed in favor of doing it after parsing with an AST transform (which is not yet complete).</li><li>A new parsing context allocates memory for AST nodes using an LLVM BumpPtr allocator (memory pool). This design is similar to the one used in Clang.<br /></li><li>We've also adopted Clang's use of a common iterator class for children of AST nodes</li><li>Identifiers in the AST are now allocated from a common StringPool, which saves memory by storing only one unique string in memory that common identifiers can point to.</li></ul>The next exciting development relates to another open source PHP compiler effort, <a href="http://phpcompiler.org/">phc</a>. We've been working with the project's maintainer, Paul Biggar, to see how we can combine efforts towards common goals. The phc project itself has a different philosphy than Roadsend PHP, namely that it uses the PHP runtime rather than a custom one. This lets it take advantage of all of the current PHP extensions and maintains very good compatibility with Zend PHP, at the expense of being limited by Zend engine/runtime design decisions.<br /><br />However, there's room for cooperation on the frontend. The phc authors have put much effort into parsing and source level analysis of PHP (Paul recently completed his PhD thesis on the subject). We're currently investigating how we can take advantage of this analysis in Roadsend PHP. The goal would be to use the same (or similar) IR structure and reuse the applicable passes that analyze and optimize PHP source, allowing us to generate more efficient low level LLVM IR.<br /><br />More on these developments soon!Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com1tag:blogger.com,1999:blog-7495993905259338393.post-56337217807477519282009-11-05T15:33:00.003+01:002009-11-05T15:48:34.181+01:00Big IntegersThe runtime recently underwent its first change that is not 100% compatible with Zend based PHP: arbitrary precision integers.<br /><br />In Zend PHP, an integer (represented as a long, which is generally 32 bits or 64 bits big, depending on the platform) will overflow to a float. This happens when you specify a large literal integer in the source code, or upon certain arithmetical operations.<br /><br />We've decided to integrate seamless "bignums" into Roadsend PHP instead. This means you can specify an integer as big as you want (only limited by memory), and that arithmetical operations will yield accurate results even when the numbers get large.<br /><br />Internally, integers that can fit into a hardware word are still represented this way, for speed. They will convert to a bignum (using GMP library) only when necessary.<br /><br />Zend PHP 5.x and current 6 has support for this now only through the GMP or BC math extensions. This is awkward since you have to juggle the numbers as resources. But, according to <a href="http://wiki.php.net/summits/pdmnotesmay09">recent developer notes</a>, this same functionality may end up in Zend PHP at some point as well.<br /><br />So, either we're a little ahead of the curve, or we're diverging slightly. Either way we think this is a win for the language as a whole.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com3tag:blogger.com,1999:blog-7495993905259338393.post-1214851456808451932009-09-17T04:12:00.003+02:002009-09-17T04:42:46.421+02:00Summer WorkWe've had a few contributions over the last few months, and I've been able to get some work done on the Error Manager as well.<div><br /></div><div><span class="blsp-spelling-error" id="SPELLING_ERROR_0">Corni</span> contributed parsing and code generation support for basic if-else blocks. He's also commented a bit and cleaned up some of the code generation. In addition, he's setup a git repository that updates from the main <span class="blsp-spelling-error" id="SPELLING_ERROR_1">svn</span> trunk. If you're interested in using that, it's available at <a href="http://gitorious.org/rphp/rphp">http://gitorious.org/rphp/rphp</a></div><div><br /></div><div>I've added the Error Manager class to the <span class="blsp-spelling-error" id="SPELLING_ERROR_2">runtime</span>, which is in charge of:</div><div><ul><li>Keeping track of current source location and <span class="blsp-spelling-error" id="SPELLING_ERROR_3">PHP</span> stack (when available)</li><li>Handling <span class="blsp-spelling-error" id="SPELLING_ERROR_4">PHP</span> <span class="blsp-spelling-error" id="SPELLING_ERROR_5">runtime</span> warnings, notices, fatal errors and exceptions</li><li><span class="blsp-spelling-error" id="SPELLING_ERROR_6">PHP</span> level, user defined error and exception handlers</li></ul><div>It's also used by the target system for emitting warnings and notices. This is because the compiler will be called from the <span class="blsp-spelling-error" id="SPELLING_ERROR_7">runtime</span> (during e.g. include or <span class="blsp-spelling-error" id="SPELLING_ERROR_8">eval</span>) and needs to be able to redirect its output to the current <span class="blsp-spelling-error" id="SPELLING_ERROR_9">runtime</span> instance.</div><div><br /></div><div>When a <span class="blsp-spelling-error" id="SPELLING_ERROR_10">runtime</span> error occurs, a C++ exception is thrown. For this to work in the <span class="blsp-spelling-error" id="SPELLING_ERROR_11">JIT</span>, it has to catch a C++ exception above the <span class="blsp-spelling-error" id="SPELLING_ERROR_12">JIT</span> engine code when thrown by the <span class="blsp-spelling-error" id="SPELLING_ERROR_13">runtime</span> that gets called from the generated <span class="blsp-spelling-error" id="SPELLING_ERROR_14">LLVM</span> IR. We had a tough time getting this to work, and ended up making a new sandbox for simplified test cases in which to test this (and in the future, other) bits of code. </div><div><br /></div><div>In the end it also led to porting the code to the latest (and as of this post, unreleased) version of <span class="blsp-spelling-error" id="SPELLING_ERROR_15">LLVM</span>: 2.6. It's in <span class="blsp-spelling-error" id="SPELLING_ERROR_16">pre</span>-release and should be available soon, however. Unfortunately this means the current trunk doesn't support 2.5, but it did fix our issues with exceptions.</div><div><br /></div><div>I'll be attending the <a href="http://llvm.org/devmtg/2009-10/"><span class="blsp-spelling-error" id="SPELLING_ERROR_17">LLVM</span> Dev Meeting</a> in a few weeks, which I'm really looking forward to. If you're going or will be in the area and want to chat about <span class="blsp-spelling-error" id="SPELLING_ERROR_18">rphp</span>, let me know!</div><div><br /></div><div>Finally, I've setup a mailing list for rphp development. You can find it at <a href="http://mail.roadsend.com/mailman/listinfo/rphp-devel">http://mail.roadsend.com/mailman/listinfo/rphp-devel</a></div><div><br /></div></div>Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com2tag:blogger.com,1999:blog-7495993905259338393.post-15415319431565616952009-07-20T21:02:00.002+02:002009-07-20T21:11:10.331+02:00On TargetWork has progressed lately on the target system. These are a set of classes used to convey information, such as configuration or command line options, to various parts of the backend that do the real work - such as compiling or JITing a single script - and also to the runtime. This paves the way for a very modular, dynamic driver system which will be necessary to use rphp from as many different front ends as are planned. For example, we're planning on supporting at least:<br /><br /><ol><li>Generating native (static file) binaries and libraries<br /></li><li>Just In Time compilation and execution of command line scripts</li><li>A web interface to the JIT, which you can fire up and point to the root directory of your PHP application<br /></li><li>Library interface to the analyzer, which can be called from e.g. an IDE</li></ol><br />Speaking of the analyzer, we've also had some progress there as well. More on that soon.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com3tag:blogger.com,1999:blog-7495993905259338393.post-52200861027839717382009-06-02T15:52:00.003+02:002009-06-02T16:06:04.802+02:00UptimeI've recently picked up a Mac Mini, my first move into the mac world. Of course one of my first tests was to see if Raven would compile. It didn't, but I was surprised at how little work it took to make it (hats off to cmake, macports, and llvm).<br /><br />I will now be testing builds on linux x86 and x86_64, and OSX.<br /><br />In addition to committing a patch that fixes the build on OSX, I've updated the tree to include the latest <a href="http://www.benhanson.net/lexertl.html">lexertl</a> release, as well as removing the "driver" library, which was unnecessary and integrated into the IR library.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com6tag:blogger.com,1999:blog-7495993905259338393.post-16269553706021249312009-05-14T16:26:00.002+02:002009-05-14T16:28:49.527+02:00DowntimeHi Folks, just a quick post to let you know that the project has <span style="font-weight: bold;">not </span>been abandoned, I've simply spent the last month moving continents and doing some design work in the process. Expect some progress and commits soon.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com4tag:blogger.com,1999:blog-7495993905259338393.post-13332945560152732922009-04-04T09:14:00.004+02:002009-04-04T11:27:51.496+02:00March MadnessMarch has come and gone. and rphp advanced a little more.<br /><br />LLVM 2.5, released in early March, is now supported. As we ended up in the release notes (thanks Chris!), a few people were interested in trying to compile an early version. To make this easier, we switched to using environment variables so that rphp can find its library files, and updated the docs on the wiki.<br /><br />A new rphp specific test suite was also added. This is based on the pcc (classic) testsuite. It only contains two tests so far, but does test both JIT interpreted and native compiled binaries. This will help prevent regressions as the code undergoes heavy development.<br /><br />The biggest item that made it in to code generation was basic user defined functions. These now work up to arity 5 (excluding passing by reference, type hints, defaults... ).<br /><br />Attention also turned back to the runtime. Hash tables now take unicode keys. We switched back to using the ICU UnicodeString as the main unicode string object in the runtime. The header files went through a large cleanup, and received some much needed documentation (more left to do here). Some basic benchmarks were added to examine the speed of runtime variable creation and destruction.<br /><br />x86-64 is now supported and tested in the build, runtime, and compiler. Both JIT and native compilation work.<br /><br />Finally, the ASIO networking library headers and a sample HTTP server were added as the base for the microserver and HTTP frontend.<br /><br />In other news, the <a href="http://code.google.com/p/unladen-swallow/">unladen-swallow</a> project was announced. This is good news for rphp, as they plan to work on several projects that Roadsend PHP can benefit from, including a new garbage collector and thread-safe code generation in LLVM.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com1tag:blogger.com,1999:blog-7495993905259338393.post-48257025222113164232009-02-28T17:13:00.002+01:002009-02-28T17:35:26.149+01:00SquawkAfter a light January, rphp saw quite a few improvements in February.<br /><br />We've switched the lexer from the beta Boost::Spirit2 to Ben Hanson's Boost::Lexer submission (<a href="http://www.benhanson.net/lexertl.html">lexertl</a>). They are actually related, as Spirit2 uses lexertl itself, but as we weren't using the parse functionality of Spirit (we're using <a href="http://en.wikipedia.org/wiki/Lemon_Parser_Generator">lemon</a>), there was no reason to endure the extra compile time overhead (it's heavily template based) and complexity.<br /><br />Speaking of the lexer, it now properly tokenizes (nearly) all PHP tokens, in any charset. Proper double quote string parsing is yet to be completed, however.<br /><br />Literal array support was added to the parser and code generator, which means all the basic types (null, int, float, bool, string, unicode string, array) have some support.<br /><br />Some other miscellaneous advances include implementing var_dump and initial support for if blocks and function declarations in the parser.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com2tag:blogger.com,1999:blog-7495993905259338393.post-76975732484722732502008-12-02T14:31:00.003+01:002008-12-02T14:53:23.139+01:00Project RavenWe've been working for a few months now on a project that we're ready to take the wraps off of a bit. It's ambitious, but Roadsend PHP always has been.<br /><br />We've started a complete rewrite of Roadsend PHP. The rewrite will address some of the problems we've had with the old code base, and hopefully open up the project to a wider range of programmers who would like to contribute.<br /><br />The new code base is written in C++ and uses LLVM for code generation. LLVM stands for Low Level Vritual Machine and provides a complete set of tools that allow us to implement a highly optimized code generator, JIT and static (native) compiler. For more information, see the <a href="http://llvm.org">LLVM website</a>.<br /><br />We're also relying on the Boost C++ libraries. These are well know, portable, peer-reviewed libraries providing lots of the low level functionality needed for the runtime. See the <a href="http://boost.org">Boost website</a> for details.<br /><br />Rounding out the major third-party libraries we've chosen is ICU: <a href="http://icu-project.org/">International Components for Unicode</a>. This library provides all the necessary tools for full unicode and internationalization support.<br /><br />Some of the goals of this rewrite include:<br /><ul><li>Highly optimized code generation</li><li>Full unicode support</li><li>Latest PHP language features (namespaces, lamda functions, etc)</li><li>Clean, easy runtime API</li><li>Portability<br /></li></ul>In the current codebase, most of the major components are in place. Simple scripts can be compiled statically (native binaries) or run through the JIT. Full unicode parsing is in place. Most major runtime components have been started.<br /><br />You can checkout or browse the codebase at the new development site:<br /><a href="http://code.roadsend.com/rphp">http://code.roadsend.com/rphp</a><br /><br />We're looking for contributors so if you'd like to help, please speak up! You can find us in #roadsend on FreeNode IRC.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com6tag:blogger.com,1999:blog-7495993905259338393.post-45172303207082100852008-10-04T07:51:00.003+02:002008-10-04T09:17:25.258+02:00How It StartedRecently, while writing a talk proposal for Roadsend PHP, I wrote down a brief history of how it got started. On a new blog intended to discuss the development and future of Roadsend PHP, it seems appropriate to start with a recap of the past. So, here it is:<br /><br />-<br /><br /><a href="http://www.roadsend.com/">Roadsend, Inc.</a> is a programming and hosting company founded by Jon Michel and <a href="http://www.linkedin.com/in/weyrick">Shannon Weyrick</a> and based in Fishkill, NY that has been in business since 1999. Roadsend has used <a href="http://www.php.net/">PHP</a> as its main server-side programming language since before the release of <a href="http://en.wikipedia.org/wiki/PHP_programming_language#Release_history">PHP 4.0</a>.<br /><br />Roadsend released its first open source project, the <a href="http://freshmeat.net/projects/sitemanager/">PHP SiteManager</a> Web Framework in 2001. Roadsend SiteManager and the succeeding Roadsend Portal CMS have been in development and use for all company projects since that time.<br /><br />The idea for Roadsend PHP began life at Roadsend in 2002 while investigating the creation of a Zend extension for speeding up Roadsend SiteManager by replacing some of the functionality with equivalent C code. Roadsend employee Tim Daly, Jr. (now at Yahoo) had the alternative idea of creating our own implementation of PHP that would compile the existing PHP version of SiteManager to native binary.<br /><br />For a time, both the C extension and the compiler were developed simultaneously. However it was decided to abandon the C extension and instead focus all effort on the new compiler implementation, which (it was hoped) would not only fulfill the desire for more speed but also offer functionality that the original PHP could not offer.<br /><br />Over the next several years we managed to develop a full implementation of the PHP language, including native compiler, interpreter, runtime, extensions, debugger, and full blown Windows IDE. It could do things the original PHP couldn't, like compile a site to a single FastCGI binary, or with an embedded web server so that it could serve itself.<br /><br />Roadsend PHP was originally offered as a commercial (closed source) solution. It was re-released as an open source project in 2007 under the GPL and LGPL licenses.<br /><br />-<br /><br />Of course, it hasn't yet taken over the PHP world, and there are several reasons for that. We intend to address these reasons, and that will be the focus of this blog. Stay tuned.Shannon Weyrickhttp://www.blogger.com/profile/10122571202251047919noreply@blogger.com4