Performance Testing I

Introduction

Speediness (execution) is one of the principles for RawDev. In order to both improve the execution time of your code as well as getting a better feel for how your coding impacts performance the RTimer object is a useful tool. The RTimer object is very simple and useful for simple situations. Examples of more advanced tools are xdebug profiler data and apache "ab" which tests performance under load.

How it works

The RTimer tool works just like a stopwatch. You can start the timer and take lap times. Lap times can be associated with labels. You can display an overview of all lap times and the total time. All methods are static so that time measurements can be taken anywhere without the use of global variables.

Example

<?php

require_once("rawdev/RawDev.php");
require_once(RAWDEV_LIB.'/Util/Timer.php'); 

RTimer::start(); # starts the timer

usleep(123);

RTimer::lap('sleep'); # records a lap time with label 'sleep'

RTimer::display(); # displays the lap times

?>

Results

php ~/rawdev/samples/Util/timer.php 
          sleep: 0.000248sec
          Total: 0.000248sec

Conclusion

I personally use this library all the time. It helps me become a better programmer by understanding which statements take longer then others. The timer is extremely easy to use but does not take performance testing under heavy load into account. Other tools such as the xdebug profiler are useful as well but they take more time to setup.

Finally, executing timer functions take time themselves (< ~.0001sec) so you want to (a) just be aware of that, (b) execute a piece of code multiple times (e.g. 100 times) to get a more accurate picture.

Links

• RTimer API Documentation
• Download

Querying the Apache log using RawDev

Introduction

The RawDev (data) model is designed to query any relational data in a uniform way. In this example I will show this how that works querying the Apache log.

Synopsis

<?php
RResource::construct('ApacheLog', './access_log')->setFilter("request_method=''")->copy()->sort('remote_host')->display(); 
?>

Read an apache log, copy the filtered results (invalid requests) to memory, sort it by IP Address, and display the results.


The results

Three Paragraph RawDev Model Theory

The Apache log relates to a Resource in the RawDev model. Resources are a set of records that can reside anywhere. Just like the Apache log that is located as a file on a file system.

The different natures of a resource in RawDev.

The nature of a resource is either: (1) Traversable, (2) Array Accessible, or (3) both.

• Traversable: A first record, next record, and last record are defined. Examples are: Apache log, csv file, and MySql table. Note t
• Array Accessible: There are one or more primary/unique keys through which a specific recordcan be retrieved (*). Examples are: Wikipedia words, Alexa domain statistics, and Oracle Table.
• Both: Any relational database table such as PostgreSQL, SQLite, MS SQL Server offers both Traversable as well as Array Accessible access.

(*) Note that traversable resources such as the Apache log could be "array accessible" by line number, however in this case this is not useful.

In our example the Apache log resource is "Traversable" but not "Array Accessible". We can easily go to the first record (open file), read the next record (read) and detect the last record (end of file). However, we cannot naturally find a specific record based on a unique key. First, the log does not have a unique key. Second, even if it had, we can't find it in the file efficiently (without doing a full resource scan).

The nature of a resource immediately impacts the kinds of actions you can do on the resource as displayed in the table below:

Traversable	Array Accessible	Both
Filter	Get	Sort
Page	Update?
Select/Copy	Delete?
Insert?	Insert?

A resource that is both traversable and array accessible inherits all the actions and can also be sorted. The question mark indicates that it's possible if the resource supports it.

Note that a traversable resource such as a csv file cannot be sorted, updated, or deleted. An easy way to get around this is to copy a filtered selection or all records to a resource (such as memory) that does support these actions and then copy the results back (if needed).

The Apache Log plugin

The Apache Log is a traversable resource and therefore needs to implement 4 methods:
(1) The constructor with specific properties to the resource

(2) The open method that opens the resource

(3) The fetch method that reads and returns the next records. A record is a hash with the column name as the key. Returns null if end of resource.

(4) The close method that closes the resource.

class RApacheLog extends RQuerySupportResource {

  var $fileName;
  var $format;

  var $titles;
  var $handle;

  function __construct($fileName, $format="%h %l %u %t \"%r\" %>s %b", $timeZone='America/New_York') {
    $this->fileName = $fileName;
    $this->format = $format;
    $this->titles= RApacheLogParser::getTitles($format);
    date_default_timezone_set($timeZone);
  }

  function fetch() {
    if ($line = trim(fgets($this->handle))) return RApacheLogParser::parseLine($this->titles, $line);
  }

  function open() {
    if (!($this->handle = fopen($this->fileName, "r"))) throw new RException('model_csv_cannot_open', 'Cannot open file [%s].');
  }

  function close() {
    fclose($this->handle);
  }

}

Above you can see the implementation for the ApacheLog resource. The constructor accepts the file path, the log format string (e.g. "%h %l %u %t \"%r\" %>s %b"), as well as the timezone as arguments (it will convert the web access date time to the timezone you specify).

Note that the plugin relies on a ApacheLog parser that (a) converts the format string into titles and (b) converts a log line (string) into a hash with the titles as keys and the line values as values.

The Example

require_once('rawdev/RawDev.php');
require_once(RAWDEV_LIB.'/Model/Model.php');
require_once(RAWDEV_LIB.'/Model/plugins/apache/ApacheLog.php'); 

$apache = new RApacheLog(dirname(__FILE__).'/access_log'); # creates the apache resource
$apache->setFilter("request_method=''"); # sets the filter to suspicious requests

$memory = $apache->copy(); # copies the filtered resource to memory (default resource)
$memory->sort('remote_host'); # sorts the results by IP address
$memory->display(); # displays the results

?>

This example shows (a) the creation of the apache resource and how the filter is set (think where clause) and (b) how the filtered results are copied into memory, sorted and displayed. The results are displayed below. Note that the copy command accepts any resource that supports it. By default the Memory Resource is chosen.

The example was executed on a log file just shy of 2 MB. Note that during the copy process all records are actually scanned and filtered, this took about 3 seconds on a Macbook.

Conclusions

If you do apache log processing, this can be a useful utility to (a) quickly scan a file or (b) store the results in a persistent database for ad-hoc queries. Of course this can all be done using RawDev. These ad-hoc queries can easily be exported to a database table, csv file, excel database using one line of code.