Sunday, March 14, 2010

Performance Testing I

Introduction


Speediness (execution) is one of the principles for RawDev. In order to both improve the execution time of your code as well as getting a better feel for how your coding impacts performance the RTimer object is a useful tool. The RTimer object is very simple and useful for simple situations. Examples of more advanced tools are xdebug profiler data and apache "ab" which tests performance under load.

How it works


The RTimer tool works just like a stopwatch. You can start the timer and take lap times. Lap times can be associated with labels. You can display an overview of all lap times and the total time. All methods are static so that time measurements can be taken anywhere without the use of global variables.

Example

<?php

require_once("rawdev/RawDev.php");
require_once(RAWDEV_LIB.'/Util/Timer.php'); 

RTimer::start(); # starts the timer

usleep(123);

RTimer::lap('sleep'); # records a lap time with label 'sleep'

RTimer::display(); # displays the lap times

?>

Results

php ~/rawdev/samples/Util/timer.php 
          sleep: 0.000248sec
          Total: 0.000248sec

Conclusion


I personally use this library all the time. It helps me become a better programmer by understanding which statements take longer then others. The timer is extremely easy to use but does not take performance testing under heavy load into account. Other tools such as the xdebug profiler are useful as well but they take more time to setup.

Finally, executing timer functions take time themselves (< ~.0001sec) so you want to (a) just be aware of that, (b) execute a piece of code multiple times (e.g. 100 times) to get a more accurate picture.

Links

Sunday, March 7, 2010

Querying the Apache log using RawDev

Introduction


The RawDev (data) model is designed to query any relational data in a uniform way. In this example I will show this how that works querying the Apache log.

Synopsis

<?php
RResource::construct('ApacheLog', './access_log')->setFilter("request_method=''")->copy()->sort('remote_host')->display(); 
?>
Read an apache log, copy the filtered results (invalid requests) to memory, sort it by IP Address, and display the results.


The results

Three Paragraph RawDev Model Theory


The Apache log relates to a Resource in the RawDev model. Resources are a set of records that can reside anywhere. Just like the Apache log that is located as a file on a file system.

The different natures of a resource in RawDev.

The nature of a resource is either: (1) Traversable, (2) Array Accessible, or (3) both.
  • Traversable: A first record, next record, and last record are defined. Examples are: Apache log, csv file, and MySql table. Note t
  • Array Accessible: There are one or more primary/unique keys through which a specific recordcan be retrieved (*). Examples are: Wikipedia words, Alexa domain statistics, and Oracle Table.
  • Both: Any relational database table such as PostgreSQL, SQLite, MS SQL Server offers both Traversable as well as Array Accessible access.

(*) Note that traversable resources such as the Apache log could be "array accessible" by line number, however in this case this is not useful.

In our example the Apache log resource is "Traversable" but not "Array Accessible". We can easily go to the first record (open file), read the next record (read) and detect the last record (end of file). However, we cannot naturally find a specific record based on a unique key. First, the log does not have a unique key. Second, even if it had, we can't find it in the file efficiently (without doing a full resource scan).

The nature of a resource immediately impacts the kinds of actions you can do on the resource as displayed in the table below:

TraversableArray AccessibleBoth
FilterGetSort
PageUpdate?
Select/CopyDelete?
Insert?Insert?

A resource that is both traversable and array accessible inherits all the actions and can also be sorted. The question mark indicates that it's possible if the resource supports it.

Note that a traversable resource such as a csv file cannot be sorted, updated, or deleted. An easy way to get around this is to copy a filtered selection or all records to a resource (such as memory) that does support these actions and then copy the results back (if needed).

The Apache Log plugin

The Apache Log is a traversable resource and therefore needs to implement 4 methods:
(1) The constructor with specific properties to the resource

(2) The open method that opens the resource

(3) The fetch method that reads and returns the next records. A record is a hash with the column name as the key. Returns null if end of resource.

(4) The close method that closes the resource.

class RApacheLog extends RQuerySupportResource {

  var $fileName;
  var $format;

  var $titles;
  var $handle;

  function __construct($fileName, $format="%h %l %u %t \"%r\" %>s %b", $timeZone='America/New_York') {
    $this->fileName = $fileName;
    $this->format = $format;
    $this->titles= RApacheLogParser::getTitles($format);
    date_default_timezone_set($timeZone);
  }

  function fetch() {
    if ($line = trim(fgets($this->handle))) return RApacheLogParser::parseLine($this->titles, $line);
  }

  function open() {
    if (!($this->handle = fopen($this->fileName, "r"))) throw new RException('model_csv_cannot_open', 'Cannot open file [%s].');
  }

  function close() {
    fclose($this->handle);
  }

}

Above you can see the implementation for the ApacheLog resource. The constructor accepts the file path, the log format string (e.g. "%h %l %u %t \"%r\" %>s %b"), as well as the timezone as arguments (it will convert the web access date time to the timezone you specify).

Note that the plugin relies on a ApacheLog parser that (a) converts the format string into titles and (b) converts a log line (string) into a hash with the titles as keys and the line values as values.

The Example


require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Model/Model.php');
require_once(RAWDEV_LIB.'/Model/plugins/apache/ApacheLog.php'); 

$apache = new RApacheLog(dirname(__FILE__).'/access_log'); # creates the apache resource
$apache->setFilter("request_method=''"); # sets the filter to suspicious requests

$memory = $apache->copy(); # copies the filtered resource to memory (default resource)
$memory->sort('remote_host'); # sorts the results by IP address
$memory->display(); # displays the results

?>

This example shows (a) the creation of the apache resource and how the filter is set (think where clause) and (b) how the filtered results are copied into memory, sorted and displayed. The results are displayed below. Note that the copy command accepts any resource that supports it. By default the Memory Resource is chosen.

The example was executed on a log file just shy of 2 MB. Note that during the copy process all records are actually scanned and filtered, this took about 3 seconds on a Macbook.



Conclusions

If you do apache log processing, this can be a useful utility to (a) quickly scan a file or (b) store the results in a persistent database for ad-hoc queries. Of course this can all be done using RawDev. These ad-hoc queries can easily be exported to a database table, csv file, excel database using one line of code.

Saturday, February 27, 2010

Mapping an Object Interface to Relational Data

Introduction

Storage of data as well as access to data in RawDev is facilitate by the data aspect or model (such as in MVC). The model is divided into three separate layers. The first layer organizes raw data into uniform entities and relations. The second layer adds security to the data by providing a mechanism to define access to data and enforcing these rules. The third layer maps entities and relationships to an object oriented interface which makes it easy to deal with nested structures and apply business logic using object oriented programming constructs.



The layers in the RawDev model (the data aspect)

Relational DataAccess ControlsObject Interface
Entities, RelationshipsUsers, Groups, RolesNested Data Structures
SQL (across data sources)Authenticated Sessionsadditional Business Logic
XML Data DefinitionRow/Column Level Access 
Caching, Copying, SynchronizingLogging 
 LDAP Integration? 
Overview of the three layers

Relational Data


Chances are that you primarily use a relational database such as MySql to store your data. RawDev provides uniform access to your favorite relational database as well as any other data source. Access to other data sources is handled by plugins. You can write a simple plugin that lists the files in the file system or hooks into an online data feed. After defining the structure of your data source in XML (think create table) you can access it and join it to any other data source using regular SQL. In addition, RawDev will provide mechanisms to cache data locally, copy entire data sets as well as synchronize data.

SELECT b.url, b.title, p.latest_post, a.ranking
FROM mysql:blog b, rss:posts p, alexa:ranking a
WHERE b.url=p.url and b.domain=a.domain
Example of a join across data sources listing 10 locally stored blog urls with their latest post and the alexa ranking on the blog domain.

Access Controls


RawDev has powerful ways to provide data access to users . RawDev provides tools to manage users, roles, organizational groups, and provide a powerful interface to grant privileges to these entities. RawDev maps and enforces these privileges in the back-end (e.g. read/write, entity, row level, and column level access). Individual users access the data using authenticated sessions. Optionally, access is reported in a central log. It is on my wishlist to provide LDAP integration (after the summer).

Object Interface


The third layer maps the relational structure to an object interface. In this layer, RawDev provides additional data access tools as well as the ability for programmers to extend this. For example: say a user profile consists of many entities such as degrees, languages etc. Using this layer you have powerful tools to save the entire profile with little effort, updating only the child records that were actually changed. Further more, additional business logic can be implemented in this layer. As simple as a uniform way to display full names or as complex as the order in which records across entities can be stored given a specific business process.




Storing a form with a nested data structure becomes a piece of cake in the RawDev Object Interface.

When do I have access to this?


Although the existing production applications such as Find a Psychologist and Fodius.com enjoy significant parts of the RawDev model, the open source version is estimated to have the full model functionality in the summer of 2010.

Philadelphia,
Raymond Bokenkamp.



Raymond Bokenkamp, MBA, MS is the founder of Bokenkamp Consulting. He holds a Masters degrees in Computer Science from the University of Twente in the Netherlands as well as an MBA from LeBow School of Business, Drexel University. Raymond has over ten years in developing programming frameworks in academic settings as well as private companies. This year he decided to release his framework RawDev as open source. RawDev is a balanced approach to sustainable programming in PHP. He started his first company in 1991, a screen-printing business, which is still thriving today and recently expanded from the Netherlands to Poland. He is now bringing his entrepreneurial spirit to provide web solutions that make a difference. Raymond loves to do yoga, write blog posts, compete in city chases, and hike in the Wissahickon Park in Philadelphia with his wife, Lisa, and dog, Jax.

Wednesday, February 24, 2010

One-stop-shop preg.

Introduction

If you use perl regular expressions on a regular basis then you probably have come across the following lines of code extensively:

<?php

if (preg_match($expression, $subject, $matches)) {
$iCareAbout = $matches[1];
}
?>

Occasionaly you also needed to replace the first matched instance. Which if you also wanted to check if matches were replaced and what the matches are is more cumbersome:

<?php

if (preg_match($expression, $subject, $matches)) {
$iCareAbout = $matches[1];
preg_replace($expression, $replace, $subject, 1);
}
?>

For both of these instances the RawDev preg function looks as follows:

<?php

$iCareAbout = RUtil::preg($expression, $subject); # case 1
$iCareAbout = RUtil::preg($expression, $subject, $replace); # case 2

?>


In addition, the RawDev preg function can match/replace all instances. Lastly, it can handle an array of strings as the subject of which only the strings that matched were returned.

API

static mixed function preg($regex, $subject, $replace, $all, $alwaysArray)

Returns the matches of a perl regular expression in an array (getting rid of the unnecessary first element that preg_match generates) ; the result is returned in a scalar if one variable is matched (\1) and alternatively in array. The matches can also be replaced.
$regexstringThe full regular expression (including options such as "/.../ice").
$subjectmixedThe subject is a string or an array of strings that is subject to modification (passed in by reference).
$replaceboolThe optional replacement string (accepts variables such as "1").
$allboolWhether one or all instances should be matched/replaced (default: all=false).
$alwaysArrayboolBy default when only one variable is matched, it is returned as a scalar (e.g. by default"/ (a) /" is returned as scalar, "/ (a) (b) /" as an array). Setting this to true will always return the matches as an array.
returnsmixedThe match or matches. This can be a string, an array of strings or an array of an array of strings (match all). Null when no matches.


Example


This example works on the string: "[USA] Gold, [Netherlands] Silver, [Canada] Bronze"

It matches/replaces the first country and then all countries.

The result is:



<?php

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Util/Util.php');

$str = "[USA] Gold, [Netherlands] Silver, [Canada] Bronze";

$str1 = $str;
$str2 = $str;
$str3 = $str;
$str4 = $str;

# find first country
$result1 = RUtil::preg("/\[(.*?)\]/", $str1);
# find and replace first country
$result2 = RUtil::preg("/\[(.*?)\]/", $str2, "\\1");
# find all countries
$result3 = RUtil::preg("/\[(.*?)\]/", $str3, NULL, true);
# find and replace all countries
$result4 = RUtil::preg("/\[(.*?)\]/", $str4, "\\1", true);

?>

Conclusion

Using RUtil::preg code will reduce the amount of code you produce. Which will, in the long run, save (a) programming hours, (b) reduce bugs and (c) increase execution performance (less code to put in memory). The subject is changed by reference. This can be annoying because you need to assign a temporary var if you don't want to change the subject. Always use preg_replace when you don't care about returning matches.

Links

RUtil API Doc

Saturday, February 20, 2010

It's not a question whether you should unit test but how much!

Introduction


Unit testing matters because it will make your code more reliable. This is especially true when code needs to be updated as time goes along. Unit testing takes discipline. It is tempting to just hack a result together and use ad-hoc testing. However, by recording these ad-hoc tests in actual unit tests you will be able to run them over and over again. It is not a question of whether you should do unit testing. Rather the question is: "What's the level of detail you are going to do unit testing?". Keep it basic, especially in the beginning. Unit tests are great for diagnostics and make your programming efforts more sustainable in the long run.

From personal experience, I can tell you how many times I have quickly build/hacked applications together. Weeks later, when I had to change something basic, this could take hours or even a day because the change I made caused other parts to fail and it was hard to diagnose. Unit testing is a skill that can help you get to the next level as a programmer as well as an organization. It will take effort and persistence but when done in the right amount you should be able to reap benefits in weeks.

RawDev has tried to make unit testing as streamlined and easy to do as possible. There are 3 aspects to unit testing in RawDev: (1) a unit test, (2) a component test (the test classes that you will write), and (3) the unit test binary that runs all, some or one unit test.

Unit Test


Overview

Code is organized in functions. These functions reside in objects or are static class functions. In addition, functions can interact with their environment.



Function
Usually an object function or a static class function. Functions can interact with their environment.
Input A function has zero or more input parameters. In rare cases these parameters defined as references in which case they can be altered by the function.
Output A function has one output parameter which could be an array.
Exception Usually based on incorrect input or failing of a resource, an exception can be thrown, which will interrup the regular flow of the code.
Object & global state Object functions can modify the state of object variables. In addition, any function can modify class variables and other global state such as databases, files etc.

Example 1


A unit test typically tests either correct output of a function or verifies if the expected exception was raised. In addition one or more asserts can be done after the function was executed. Asserts are typically tests of the object and global state. Don't worry about how results of tests are displayed, I will get to that in the next section. RawDev uses fluid API calls for usability.

Let's look at some examples. First a real basic example of a function that calculates the sum of 2 or more inputs. Each test has a title for diagnostics but as we will see later the title will set to the test function that implements the test.

<?php

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Test/Unit/Test.php');

function sum() {
  $params = func_get_args();
  if (count($params) < 2) throw new RException('math_too_few_params', 'Two or more parameters please.');    
  $sum = 0;   
  foreach($params as $param) $sum += $param;    
  return $sum; 
}

# test 1 
$t = RUnitTest::construct('sum')->title('Two Params')->in(1, 2)->out(3)->test();
print $t->icon;

# test 2
$t = RUnitTest::construct('sum')->title('One Params')->in(1)->exception('math_too_few_params')->test();
print "$t->icon\n";

?>

The steps of the first test are: set function, set title, set input params, set output params, test function and evaluate results.

The second function tests wheter the exception is thrown as expected.

Example 2


Let's look at a simple object test next.

<?php

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Test/Unit/Test.php');

class Car {

var $miles = 0;

function drive($miles) {
if ($miles <= 0) throw new RException('car_illegal_miles', 'Miles should be more than zero.'); $this->miles += $miles;
}

}

$c = new Car();
$f = array($c, 'drive');

$t = RUnitTest::construct($f)->title('Simple Drive')->in(10)->out()->test();
$t->assert(10, $c->miles);
print "$t->icon\n";

?>

In this example the test verifies whether the output is NULL. The assert tests wheter the "object variable" mile was set to 10.

Component Test


The three components to test in RawDev are: modules (or sub-modules), classes and functions. Each of these components has a setup routine (optional), zero or more unit tests and a cleanup routine (optional). Each Component test is implemented as a test class. Creating test classes for modules and classes is often not necessary.

Example


The example below shows all the tests for the "drive" function (see example above). Note that the unit function creates a unit test object and sets the title to the name of the function. All functions that start with test[Name] are run as unit tests. It is perfectly ok to create helper functions that do not start with "test".

<?php

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Test/Unit/Component.php');

# class from the previous example
class Car {

var $miles = 0;

function drive($miles) {
  if ($miles <= 0) throw new RException('car_illegal_miles', 'Miles should be more than zero.'); 
  $this->miles += $miles;
}

}

# this shows what your test harnass will look like
class TCarDrive extends RComponentUnitTest {

  var $car;
  var $f;

function setup() {
  $this->car = new Car();
  $this->f = array($this->car, 'drive');
}

#Drive10 will show up as the title of the unit test!
function testDrive10() {
  $t = $this->unit($this->f)->in(10)->out()->test();
  $t->assert(10, $this->car->miles);
}

#Negative: test for an exception
function testNegative() {
  $t = $this->unit($this->f)->in(-1)->exception('car_illegal_miles')->test();
}

}

#run the harnass, in real life this is done throug: bin/runit.php
$c = new TCarDrive('function');
$c->test();
$c->display();


?>

Component Test location


Test classes are saved in the "tests" directory using a specific path/name structure:

[RAWDEV_HOME]/tests/Util.php : indicates a component test for module Util.
[RAWDEV_HOME]/tests/Util/Car.php : indicates a component test for class Car (part of the Util module)
[RAWDEV_HOME]/tests/Util/Car/drive.php : indicates a component test for function drive of class Car (part of the Util module)

Module and Class Components

Module and class components are not required. In addition, they can contain setup/cleanup routines for the module or class only. Modules and classes do not have dependencies because they are expected to be tested as independent modular constructs.

Dependencies

A function may have implicit or explicit dependencies on other functions in the class. RawDev detects implicit dependencies by reading the source of the function. Explicit dependencies are added using the @dependencies notation in the function comment header (e.g. @dependencies __construct,start). RawDev (a) automatically rearranges the order that functions are tested based on these dependencies and (b) will unit test dependent functions if a depending function is tested in isolation.

Unit Test binary


[RAWDEV_HOME]/bin/runit.php is the binary that runs all unit tests it accepts zero or one argument which is the path of the component to be tested.

Examples:

run all unit tests:
[RAWDEV_HOME]/bin/runit.php

run all unit tests for module Util:
[RAWDEV_HOME]/bin/runit.php Util

run all unit tests for class Car:
[RAWDEV_HOME]/bin/runit.php Util/Car

run all unit tests for function drive:
[RAWDEV_HOME]/bin/runit.php Util/Car/drive

run one unit test named drive10
[RAWDEV_HOME]/bin/runit.php Util/Car/drive/testDrive10

One last note that when a component is tested, then all it's child components are tested as well. It's parent components are not tested. However the setup and cleanup routines are called for all it's parents.

Test Output


The output of any component test (or all tests) is divided into 3 parts: tree view, overall view, and failed test messages.

Example:
php runit.php Util

Util 
    Dumper 
        display .....
        dump ..
        formatScalar .......
        maxLength ..
        page ....
    Match 
        __construct .
        _match ...
        matchObject ...
        matchScalar .....
        matchArray .......
    Stdin ..
    Util 
        max .X.
        min ...
        nvl .......
        preg ................
        writeFile ..

..........................................X.......
......................

X: Util/Util/max:Array                     : Value [9] does not match expected value [10] for item []. in Match.php (115)
   Match.php            96
   Match.php            57
   Test.php             179

As you can see all tests succeeded except for the "testArray" unit test function of function max in the Util class. Extra information is displayed (including a simple trace) why the test failed.

Statuses:
. : OK
X : FAIL (output does not match expected)
E : EXCEPTION (unexpected exception occured)
? : NO_EXCEPTION (expected exception did not occur)

Summary


Unit Testing takes discipline. When done at the appropriate amount it will increase productivity and reliability. It is also a great diagnostic tool.

RawDev organizes unit tests into component test classes. The runit binary can test one unit test, all unit tests or any component and it's children. When a function changes the state of the object or global state then multiple assert calls can be made to verify the changed state. For this reason the object or global variables should be accessible publicly or by method. Unit tests are most effective when functions have one specific task. Doing unit testing will actually alter the way you code (this is not a commercial for suits).

Links

RUnitTest API Doc

RComponentUnitTest API Doc

PHP Magazine articel

Wednesday, February 17, 2010

Debug PHP code in an IDE in less than one hour.

Introduction


Although RawDev offers debugging tools, it can be useful to use an IDE with a debugger. I will show you how to setup PHP debugging using open source software for both command line and web page debugging.

 This example uses eclipse as the IDE and XDebug as the PHP module, note that this is not the only software available. You need root access and already have apache and php installed. The instructions are for Mac OS X / Unix but should be very similar for Windows users.

Steps

* Setup XDebug
  * Install XDebug
  * Alter php.ini
* Install Eclipse
* Hello World example
  * Create Hello World example
  * Debug as PHP Script (command line)
  * Debug as PHP Web Page

Setup XDebug

Install XDebug (easiest)

> pecl install xdebug

Install XDebug (unix / alternate)

> cd /usr/local/src wget http://xdebug.org/files/xdebug-2.0.5.tgz
> tar xvfz xdebug-2.0.5
> cd xdebug-2.0.5
> phpize
> ./configure --enable-xdebug
> make
> # mkdir /usr/local/php/modules # if it doesn't exist
> cp modules/xdebug.so /usr/local/php/modules/

Alter php.ini

Add the following lines to the php.ini :

# under the extensions ...
zend_extension="/usr/local/php/modules/xdebug.so"

# somewhere at the bottom ...
[xdebug]
xdebug.remote_enable=1

Test XDebug

* restart apache
* create a script (info.php) in your web_root folder.
* check the script and make sure XDebug is enabled see the screenshot at the bottom of document.

Install Eclipse


* download Eclipse for PHP Developers from: http://www.eclipse.org/downloads/download.php
* install it

Hello World example

Create example

* run eclipse
* >File>New>Project>PHP>PHP Project
* Use default path: "~/Documents/workspace/Hello"
* Project Name "Hello" [Next] [Finish] [Yes]
* >File>New>PHP File
* Name : index.php [Finish]


Debug as PHP Script (command line)

* >settings>PHP>PHP Executables>Add
* Name: "php" ;
   Path: "/usr/local/php/bin/php" ;
   php.ini: "/usr/local/php.ini" ;
   PHP Debugger: "XDebug"
   [OK]
* right click (in index.php source window)
* select debug as PHP Script

Debug as PHP Web Page

* Make sure the project path is visible in the webroot
* one way: create a symbolic link to project directory and
                   project directory is visible for all (e.g. chmod 755)
* test this in a browser
* >settings>PHP>PHP Servers>Add (&make default)
* Name: "php" ;
    Path: "/usr/local/php/bin/php" ;
    php.ini: "/usr/local/php.ini" ;
    PHP Debugger: "XDebug" [
    [OK]
* right click (in index.php source window)
* select debug as PHP Web Page

Conclusions

Obviously these are instructions just to get you started. The other debugging module is "Zend Debugger" and other PHP IDE's are available (e.g. Komodo). RawDev will use more of XDebug extended functionality such as profiling.

Links

* http://xdebug.org
* http://eclipse.org


phpinfo screenshot:

Sunday, February 14, 2010

When var_dump is not enough.

Introduction


Using this library you can browse the content of complex variables, one nested level at the time by using commands such as zoomin, zoomout and exit. Currently, this class should only be used for command line applications. Web browsable functionality will be added later using through firephp/firebug.

Example


By executing the snippet below you will see the following screen. At this point you can decide to exit (or zoomout), or zoom in to nested variable links [1] and [2].


<?php

require_once("rawdev/RawDev.php");
require_once(RAWDEV_LIB.'/Util/Dumper.php');

class Test {

var $a='hi';
var $b = array(1, 2);

}

$complex = array(1, 'hi', array(99, 17), 'std' => new stdClass(), new Test(), array());

RDumper::dump($complex);

?>

Conclusion

For command line application development this beats var_dump and will save you a ton of time when debugging.

Links

RDumper API Doc

Future enhancements

Color coding?

Saturday, February 13, 2010

Command line user input

Introduction

RStdin is tiny class that deals with interactive keyboard input from the command line. For testing purposes the keyboard input can be faked.

example


This example first shows regular use. The output is trimmed by default but this can be changed (see RStdin API Doc). In the second part you can see how the keyboard input can be faked for testing purposes.

<?php

require_once("rawdev/RawDev.php");
require_once(RAWDEV_LIB.'/Util/Stdin.php');

# regular use
$input = RStdin::read('Please enter input: ');
print "$input";

# testing use
RStdin::$buffer[] = 'hello'; # for testing purposes the input buffer can be set
$input = RStdin::read('Please enter input: ');
print "$input";

?>

Conclusion


This class is useful when dealing with command line applications. RawDev uses this class in the Dumper utility in which you can interactively view nested complex data structures.

Links

RStdin API Doc

Saturday, February 6, 2010

Handling errors, warnings and exceptions with one handler.

Introduction

In RawDev, all PHP errors and exceptions can be handled centrally using the Error Reporting library. In addition, RawDev gives more options of error messages than the standard PHP error_reporting level. For example, by default strict errors such as illegal variable references are returned but strict errors such as undefined array offsets are not. RawDev finds that being more strict with variable (and object property) references leads to catching programming mistakes earlier in the development process. Most RawDev users will find this class extremely useful.

Example

In the example below, I trigger several PHP errors: invalid offset, illegal variable reference and illegal object property reference. As you can see, the invalid offset is ignored by default and both other errors are thrown as RExceptions (RawDev Exceptions). Any other RExceptions can be handled by a central handler as well.

<?php

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Core/ErrorReporting.php');

RErrorReporting::singleton()->initialize();

$b = array();
if ($b[0]); # this will not trigger an error

$difficultVariable = 17;
try {
if ($difficultVaariable); # this triggers an error because the wrong variable is used.
}
catch (RException $e) {
print $e->getMessage()."\n";
}

class Test {
}

$a = new Test();

try {
if ($a->b); # this triggers an error because an invalid object variable is used.
}
catch (RException $e) {
print $e->getMessage()."\n";
}

?>

Conclusion

This is just one of these classes that as a developer you will get attached to because it forces you to code a little tighter but avoids having to debug illegal variable and object property mistakes. The central error handling is a nice bonus.

Links

Error Reporting RawDev API Doc

Saturday, January 30, 2010

Regular expressions for non-string complex data?

Introduction


RawDev is a sustainable framework that is build from the ground with unit tests. Before I get deeper into unit testing I want to introduce to you a helper library that makes validating any data structure (simple scalar or complex array structure) easy. This library is a key building block to assert functionality in RawDev's unit testing but its use does not end there. Other uses I can think of are: debugging, (non-string) validation. Perhaps you can think of more uses. As you might expect this library is also fully extendable and ample examples are available in the download.

Synopsis


<?php

$expected = 'Hello World';
$value = 'Hello World';

$obj = new RMatch($expected);
$obj->match($value);

?>

This is a silly example that you could do much simpler for scalar values. But stick with it. The RMatch utility library simply expects a $expected value (or definition) to which it then compares the target $value.

Array example

<?php
$expected = array('person' => array('name' => 'Jane', 'spouse' => array('name' => 'John', 'address' => array('city' => 'Philadelphia'))));
$value = array('person' => array('name' => 'Jane', 'spouse' => array('name' => 'John', 'address' => array('city' => 'New York'))));

$value = $expected;
RMatch::construct('RMatch', $expected)->match($value); #fluid interface
?>
This is getting more interesting because not only does RMatch compare the two arrays it also suggests where the error occurs. RMatch is equipped to compare arrays, objects and scalars. You can tell RMatch what exact flavor of matching you seek (e.g. scalar type checking). You can also do things like: check if these three attributes of an array or object match, ignore other attributes. Finally, you can extend RMatch with your own definiting of matching.

Your own matching algorithm


It is easy to extend RMatch with your own matching algorithm. In the example below, I briefly illustrate how to implement a "like match" (just like in SQL)

<?php

class RLikeMatch extends RMatch {
function matchScalar($expected, $value, $path) {
if (!preg_match("/".str_replace($expected, "%", ".*?")."/", $value)) {
throw new RException('nonlike_match_scalar', 'Value [%s] does not like-match expected value [%s] for item [%s].', $value, $expected, $path);
}
}
}

RMatch::construct('RLikeMatch', 'dog%')->match('dog person');

?>

Conclusion

The RMatch library is modular, powerful, lean and easy to use. Although it's initial use was to help assert any test result for unit testing, it can be used in many other situations such as validation and debugging. Although you are likely just to use the defaults, it is easy to change the algorithm on which complex structures are matched. In addition, any part of a complex tree is extendible to have it's own custom match capability.


Future

Using this library reminded me a lot of doing regular expressions on arrays. This idea could be expanded more in the future. A simple syntax could describe definitions such as the count, order, required, etc.

Links

RMatch API Documentation
Download RawDev

A Vision of RawDev

Imagine a scenario:
I have survey results in a Google Doc that gives me a list of websites that my users like. I have a website like http://www.alexa.com/ that provides rankings of websites worldwide. I have a table of data on websites in Oracle.

Three data sources, three disparate data sets. But looking at it, we know that there are linkages there. Imagine an application framework that allowed you to query all three, simultaneously, and seamlessly present the data to your users in a convenient front-end.

Now imagine that framework is called RawDev.

This sort of project is exactly what RawDev is designed to do: bring together independent data and give it to an end-user in a seamless presentation. You set up the datastore once, and then build apps to use that data again and again. Integrate different types, different locations, even different storage engines of data and have one interface. View all these different data-sets as one data-set in your favorite database gui application. Oh, and RawDev will throw in powerful XML based access controls as well.

You can download a beta version of RawDev this summer after a live demo at the Higher Ed Web Symposium at the University of Pennsylvania on July 21st and July 22nd. In the mean time you can get early bird access and follow the development process on this blog and http://rawdev.bokenkamp.com.

Friday, January 29, 2010

Dealing fatal errors in PHP

When a fatal error in PHP occurs, the flow of the program halts resulting in a blank screen or a system error message. Leaving your end-users confused and you, as the programmer, unaware that these fatal errors might be occuring.

RawDev offers a simple way to handle fatal errors more gracefully so that you can (a) log/email the error and (b) display an appropriate message and a path to continue so that the end-user is not staring at a system message or worse, a blank screen unsure what to do next.


How it works

PHP displays the fatal error right before it halts. RawDev reads the output buffer and detects when a fatal error occurs. That's when it calls your custom handler.

Basically all you have to do is register a fatal error handler. See example below.

Example


RFatal::setHandler('fatalErrorHandler');

hello(); # since this function does not exist it will throw a fatal error

# this area is never reached because you get a handle, but PHP is going to halt anyways

RFatal::unsetHandler(); #optional function to remove the fatal error handler

?>

Display Errors


Note that display_errors needs to be set for the RFatal class to work, which is the default in PHP. Usually this config parameter is set by the sysadmin in the php.ini file. Sometimes (typically in production) this is turned off for security reasons with non-favorable usability impact. In RawDev this needs to be turned on. The security issue no longer applies because an appropriate message to the end-user should be displayed by you, the programmer, in the fatal error handler. Typically something similar to a 404 with a simple message and a path to continue.



Conclusion

By trapping the output buffering you can actually get a handle to fatal errors, and RawDev makes it easy to do this. There are cases where you do not want to do output buffering, for example when dealing with very large output. In that case, don't use fatal error trapping.



Links

API Doc for RFatal

Wednesday, January 27, 2010

Unit Testing - Part I

Introduction




Unit testing matters because (a) your code will be more reliable, especially when your code is updated over time and (b) you will reduce programming hours over time. It is tempting to solve the problem and hack a solution together as quickly as possible, doing ad-hoc testing as you go. Taking the time to unit test may add approximately 15% of initial coding time.

A common scenario is that everything is crystal clear when the initial coding is done. However, we live in a dynamic world and things change. So when adding a feature months later, all of a sudden it becomes a lot harder to verify if everything is working properly. You may spend a full day on a silly bug that could have easily been pointed out by unit tests. This is frustrating and not sustainable over time. The investment in Unit testing will earn back those 15% of extra effort you put in initially, quickly.





How it works

Let's consider a simple black box model. A function has input parameters and as output it has the returned result. In addition, an exception can occur, in which case no output is returned. This is diagrammed below.


 

RawDev make the setup of unit tests as easy as possible by using a fluid interface so that you can typically fit a test on one line of code. The time it takes to do unit tests now solely depends on how deep you are going to test. Typically you would like to produce fewer tests that cover as many as possible scenarios. By nature certain functions require more testing based on how critical they are. Tests are just like code, you have to maintain them over time.

The fluid methods to setup your unit test are:
setTitle(string $title); #optional, but useful, the title is displayed when a test fails
setInput(mixed $param, [$param 2 ...]); #sets the input parameters
setOutput(mixed param); #sets the expected returned result
setExpectedException(string $type); # sets the expected RException type

function setInput($input[, $...]) [fluid]

function setOutput($output) [fluid]

function setExpectedException($expecteException) [fluid]

function setTitle($title) [fluid]


The final test execution method is:

bool function test()

Example


Consider a simple function that returns the sum of multiple floats or integers based on two or more input parameters. The function sum is displayed below:

<?
function sum() {
  $params = func_get_args();

  if (count($params) < 2) throw new RException('math_too_few_params', 'Two or more parameters please.');

  $sum = 0;
  foreach($params as $param) $sum += $param;

  return $sum;
}
?>

My test strategy is to quickly test 0, 1, 2 and 3 params and see if the right output is produced. See the RawDev code below. In addition, for demo purposes I include a test for the invalid output, an expected exception that didn't happen and an exception that wasn't trapped.




RFunctionUnitTest::construct('sum')->setTitle('Zero Params')->setInput()->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('One Params')->setInput(1)->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('Two Params')->setInput(1, 2)->setOutput(3)->test();
RFunctionUnitTest::construct('sum')->setTitle('Three Params')->setInput(1, 2, 3)->setOutput(6)->test();
RFunctionUnitTest::construct('sum')->setTitle('Invalid Output')->setInput(1, 2)->setOutput(4)->test();
RFunctionUnitTest::construct('sum')->setTitle('Expected Exception Incorrect')->setInput(1, 2)->setExpectedException('math_too_few_params')->test();
RFunctionUnitTest::construct('sum')->setTitle('Untrapped Exception')->setInput(1)->setOutput(1)->test();
Output:

....X?E

X: Invalid Output              : Value is [3] but should be [4].
?: Expected Exception Incorrect: Exception [math_too_few_params] was expected.
E: Untrapped Exception         : Two or more parameters please.

Conclusion

RawDev makes it easy to define the expected output (result/exception) of a function based on the input. It also has an easy way of labeling a test and displaying the results of many tests. What is not yet incorporated is (a) the output of non-scalar variables such as hashes and objects (not such a big deal) and testing of object functions that change the state of an object. These topics will be added and discussed in the near future.

Links

Exceptions
Fluid functions
Function Test API Doc

Tuesday, January 26, 2010

Exception Handling

Examples of exceptions in a web application are model or data layer errors such as "Cannot connect to the database" or user errors such as "Email format incorrect". RawDev offers a simple consistent way of raising these exceptions so that they can (a) be properly logged when necessary and (b) be properly displayed to the end user.

RawDev simply extends the existing PHP Exception with the RException object. Mainly, the added functionality is that you can raise an error with a type (e.g. "division_by_zero"). In addition you can specify a message with parameters (such as used in sprintf). This is (a) handy for the programmer and (b) allows for different languages (i18n) down the road.

Lets get to it:

Example:

require_once('RawDev/RawDev.php');

try {
  throw new RException('email_illegal_format', 'Email [%s] has an illegal format', 'test@test');
}
catch (RException $e) {
  if ($e->type == 'email_illegal_format') print $e->message."\n";
}

?>

In conclusion, RawDev offers a hybrid of using the basic PHP Exception in which you can raise different errors by integer code versus the advanced PHP capability of extending every error type Exception with it's own Exception class (e.g. class EmailIllegalForm extends Exception).

RawDev seeks to (a) avoid using integer codes as error identifies because they are hard to keep track of and (b) creating hundreds of classes for each error type.

Links:
RawDev API Doc

Comments ?

Wednesday, January 6, 2010

Introducing RawDev : a practical PHP framework

I want to let you know that I have decided to publish the framework RawDev that I am working on under the MIT license.


RawDev is a practical MVC framework for Rapid Web Development in PHP. Using RawDev you can program web applications faster in your team or by yourself.


RawDev seeks balance between the following principles: simplicity, modularity, flexibility, power, usability, speediness (execution), security, and reliability (well tested).


RawDev is also intended to be well-documented, collaborative and consistent.

Currently two production applications are using RawDev: http://HealthPanda.com and http://Fodius.com

Version 0.01 is a much more basic version (for now), it includes the error handling core library. The reason for this approach is that I will add documentation and unit tests every week before that library or module is published. The good news is that through weekly updates you will get a unique learning experience and sense of how RawDev works. The bad news is that  if you are eager to start  building web apps with RawDev you will have to wait until more modules are made available.


These weekly updates will lead to a presentation this summer at the Higher Education Web Symposium on July 21 & 22, 2010 at the University of Pennsylvania.  

RawDev can be downloaded from :
http://rawdev.bokenkamp.com

Also, expect weekly updates on:
http://blog.bokenkamp.com

Please forward this to someone you know that finds this interesting as well.

Contact me for feedback by clicking here.

--Raymond.
raymond@bokenkamp.com