How to develop WhatWeb 0.4 plugins
----------------------------------
by Andrew Horton aka urbanadventurer. MorningStar Security http://www.morningstarsecurity.com/
Revision 1.1, 29th March 2010.
Contents
=================================================
1. Introduction to WhatWeb
2. Introduction to WhatWeb plugins
General aims of a plugin
Methods to identify systems
Important files and folders
Anatomy of a plugin
3. Research background information
4. Collect samples
Website Showcases
Using Search Engines
Forums for website development with the cms
5. Analyze samples
Read the source of a couple of samples
Collect HTML and HTTP headers from samples
Remove incorrectly identified samples
Examine the samples with WhatWeb
Remove more incorrectly identified samples with the whatweb report
Use find-common-stuff to automatically identify common strings in the samples
Analyse HTTP headers and cookies
Read more HTML source
6. Review of unique patterns identified
7. Write the plugin
8. Closing notes
9. Resources
1. Introduction to WhatWeb
=================================================
WhatWeb lets you identify content management systems (CMS), blogging platforms, stats/analytics packages, javascript libraries, servers and more. When you visit a website in your browser the transaction includes many unseen hints about how the webserver is set up and what software is delivering the webpage. Some of these hints are obvious, eg. "Powered by XYZ" and others are more subtle. WhatWeb recognises these hints and reports what it finds.
WhatWeb has many plugins and needs community support to develop more. Plugins can identify systems with obvious identifying hints removed by also looking for subtle clues. For example, a WordPress site might remove the tag <meta name="generator" content="WordPress 2.6.5"> but the WordPress plugin also looks for "wp-content" which is less easy to disguise. Plugins are flexible and can return any datatype, for example plugins can return version numbers, email addresses, account ID's and more.
There are both passive and aggressive plugins, passive plugins use information on the page, in cookies and in the URL to identify the system. A passive request is as light weight as a simple GET / HTTP/1.1 request so it is suitable for large scale scanning of websites. Aggressive plugins guess URLs and request more files.
2. Introduction to WhatWeb Plugins
=================================================
Plugins are easy to write, you don't need to know ruby to make them but it helps.
General aims of a plugin
------------------------
Most plugins have a primary aim which is to identify a type of system based on signatures. The system could be a:
* Content Management System
* Javascript Library
* HTTP Server
* Application Framework
Some plugins do not have the aim to identify a specific type of system. Instead they try to give information that can be used to identify unanticipated systems or can be used for all types of websites. These plugins are:
* Title
* MD5 hash
* Meta generator tag name
* Uncommon HTTP headers
Methods to identify systems
---------------------------
There are 4 main methods to identify a CMS or web application. They are:
1. Matching patterns in the HTTP headers and HTML of a simple webpage request
2. Testing for URLs and identifying patterns in the HTML
3. Testing for URLs and recognising the MD5 hash of the HTML
4. Testing for URLs and simply noting they exist or return an HTTP status 200 code.
WhatWeb supports all 4 methods however the 1st method is the most useful in large scale scanning. It is also the most efficient by trading off knowledge for network bandwidth and time.
Support for the first method is the most developed method within WhatWeb and is discussed in detail in this document. Future development of WhatWeb will add more user friendly support for methods 2 through 4 which come under the purview of aggressive plugins.
Important files and folders
---------------------------
The important folders to plugins are:
* disabled-plugins/
* plugin-development/
* plugin-development/tests/
* plugins/
All .rb files in the plugins/ folder are loaded by WhatWeb. To disable a plugin, move it into the disabled-plugins/ folder.
The plugin-development folder contains some tools that are useful in developing plugins.
The tools are:
* find-common-stuff - This searches for common strings among a set of HTML files
* wget-list - This downloads a list of example websites
The plugin-development/tests folder contains example webpages of CMS's to study. The wget-list will create two files for each example webpage. A .html file and a .meta file.
Anatomy of a plugin
-------------------
This is a typical plugin. It identifies the Drupal framework and it's split into sections and given line numbers.
->-----------------------------------------------------------------------------------------------------------
1 Plugin.define "Drupal" do
2 author "Andrew Horton"
3 version "0.1"
4 description "Drupal is an opensource CMS written in PHP. Homepage: http://www.drupal.org"
-<-----------------------------------------------------------------------------------------------------------
Line 1. has the name. This name can be referred to on the commandline in a case insensitive way.
For example, the following works:
$ ./whatweb -pdrupal www.example.com
Line 2. has the author. Just fill in your name between the double quotes.
Line 3. contains the version number. It's up to you what number to choose.
Line 4. Contains the description. This should contain a description of what the plugin identifies that anyone can understand. It can be many lines but must start and end with double quotes.
Note that the author, version and description follow the format:
field-name field-content
On the left is the name of the variable and on the right, separated by a space is the value. This type of variable declaration isn't ruby code, it's specific to the plugins and only works for certain variable names.
The list of variable names that can be declared in a plugin in this manner are:
* author
* version
* description
* examples
* matches
->-----------------------------------------------------------------------------------------------------------
5 # hard to identify
6 #<a href="http://drupal.org"><img src="/dagboek/misc/powered-black-80x15.png" alt="Powered by Drupal, an open source content management system" title="Powered by Drupal, an open source content management system" width="80" height="15" /></a> </div>
7 # <script type="text/javascript" src="/misc/drupal.js"></script>
8 # <script type="text/javascript" src="/main/misc/drupal.js"></script>
9 # @import "/misc/drupal.css";
10 # Set-Cookie: SESS6bdd09d4debccdc3a0f49becc449e8d5=2sq674vjn6vig48e3podh3j8e2; expires=Fri, 11 Dec 2009 15:37:52 GMT; path=/; domain=.moby.com
11 # Set-Cookie: SESS9795bcd4ea70e3f846e84f29f9491636=57eafcca6400d894772a136fb5889b92; expires=Fri, 11-Dec-2009 15:38:25 GMT; path=/; domain=.save-your-future.com
12
13
14 examples %w| amnesty.org/ appel.nasa.gov/ beta.worldbank.org/ entergy.pewclimate.org/ labs.divx.com/ lindenlab.com/ littlestarprints.com moby.com/ myplay.com/ sequelnaturals.com/ teen.secondlife.com/ www.artwaves.de www.asys.com.br/ www.atomicbop.net www.cristal.com.pe/?adulto=si www.dutchbutnotfromholland.eu/ www.elespectador.com/ www.ensembles.com.ph/ www.foxsearchlight.com/index.php www.freshbrain.org/ www.icsalabs.com/ www.johnnycashonline.com/ www.journalismcenter.org/ www.jovenscriativos.com.br/ www.koalafoundation.org.au/ www.la2day.com/ www.moove.be www.mtv.co.uk/channel/flux www.mulinobianco.it/ www.multiways.com/ www.nowpublic.com/ www.pravda.lt/ www.realismssoftware.com/ www.save-your-future.com www.shock.com.co/ www.sosojuicy.com/ www.spreadfirefox.com/ www.tidningenresultat.se www.ubuntu.com/ www.universitytowers.net/ www.warnerbrosrecords.com |
15
-<-----------------------------------------------------------------------------------------------------------
Lines 5 through to 11 are comments. Each commented line must begin with a # character and this is a standard ruby way to comment code.
Line 14 is a list of example websites. The examples prefix of %w| means an array of elements separated by whitespace. The individual examples are URLs. If they are missing the http:// or https:// then http:// is assumed.
If you prefer you can list the examples like this:
examples %w|
http://www.example.com
http://www.example2.com
http://www.site.com/blah/
|
->-----------------------------------------------------------------------------------------------------------
16 matches [
17 {:name=>"/misc/drupal.js",
18 :probability=>100,
19 :regexp=>/<script type="text\/javascript" src="[^\"]*\/misc\/drupal.js[^\"]*"><\/script>/},
20
21 {:name=>"Powered by link",
22 :probability=>100,
23 :regexp=>/<[^>]+alt="Powered by Drupal, an open source content management system"/},
24
25 {:name=>"/misc/drupal.css",
26 :probability=>100,
27 :regexp=>/@import "[^\"]*\/misc\/drupal.css"/},
28
29 {:name=>"jQuery.extend(Drupal.settings,",
30 :probability=>100,
31 :text=>'jQuery.extend(Drupal.settings,'},
32
33 {:name=>"Drupal.extend(",
34 :probability=>100,
36 :text=>'Drupal.extend('}
37 ]
-<-----------------------------------------------------------------------------------------------------------
This section is a list of patterns to match against the webpage. Matches is an array and each element of the array is a hash and is surrounded by {} brackets. Notice that each pattern has a comma after it except for the last one. This is the normal ruby method of defining an array except that there is whitespace between matches and the content.
Lines 17 through 19 define the first pattern.
Line 17 defines the pattern name. The name can be anything that describes what it's matching.
Line 18 defines the probability of the pattern correctly identifying the system. It's not a real probability, instead it refers to the certainty that the match correctly identifies the system:
The probability values are:
25 = Maybe
75 = Probably
100 = Certain
Line 19 contains the pattern to match. It is a regular expression but could be any of the following list:
* regexp - Regular Expression. Standard ruby regular expression surrounded by slashes.
* text - Simple string of text surrounded by " or ' quotes
* ghdb - Google Hacking Database. This is a google-like query that supports a few parameters.
The parameters supported by ghdb are:
* inurl: - the following string is in the URL
* intitle: - the following string is between the <title> </title> tags
* filetype: - the following string is the file extension, eg. PDF, JPG, RB, etc.
* - - the following string is not matched on the page
The match used on Line 19 is regexp and the pattern is:
/<script type="text\/javascript" src="[^\"]*\/misc\/drupal.js[^\"]*"><\/script>/
The slash needs to be escaped with a backslash. That is why "text/javascript" is written as "text\/javascript". This is a standard ruby regular expression which differs slightly from regular expressions in other languages. To learn to write regular expressions visit http://rubular.com/ where you can copy & paste some HTML into the box then test out different regular expressions to see if they match.
->-----------------------------------------------------------------------------------------------------------
38. def passive
39. m=[]
40. #SESS 9795bcd4ea70e3f846e84f29f9491636 =6b74f8aff4bf7d34d181a6a380d1ec7b; expires=Tue, 15-Dec-2009 15:21:24 GMT; path=/; domain=.save-your-future.com
41. m << {:name=>"SESS Drupal Cookie", :probability=>75 } if @meta["set-cookie"] =~ /^SESS[a-z0-9]{32}=[a-z0-9]{32}/
42. m
43. end
44. end
-<-----------------------------------------------------------------------------------------------------------
Lines 38 through 43 defined the passive function. This function is called everytime the plugin is matched against a webpage.
Functions are able to access the following variables:
* @body - The HTML body
* @meta - The HTTP Headers include cookies
* @status - The HTTP status code. 200 is successful, 404 is not found.
* @base_uri - The URL
The passive plugin on line 39 creates an empty array called m. On line 42 it returns that array. The m array will either be empty or will have the sames fields as the patterns in the match array.
Line 40 is a comment which contains a sample session cookie
Line 41 adds the hash to m if the @meta array element 'set-cookie' matches the regular expression /^SESS[a-z0-9]{32}=[a-z0-9]{32}/
This regexp means a line that starts with SESS followed by 32 lowercase letters or numbers followed by the equals sign which is followed by 32 lowercase letters or numbers.
Line 44 ends the plugin which was started on line 1.
3. Research background information
=================================================
Go to the homepage of the software or CMS you are researching and learn about it.
Look for:
* Requirements, eg. the type of web server and languages it requires
* Demo sites
* Website showcases and portfolios
* Download links
* Documentation.
Some of this information will help in writing the plugin description and some will be useful in collecting samples.
The information I gathered:
* The SilverStripe homepage is http://www.silverstripe.com/
* The opensource CMS software is at http://silverstripe.org/
* Documentation of requirements is at http://doc.silverstripe.org/doku.php?id=server-requirements
* A project showcase at http://www.silverstripe.com/project-showcase/
* A demo site at http://demo.silverstripe.com/
Using the information found I wrote the following plugin description:
"SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"
Advanced hint: If you intend to make an aggressive plugin then you may wish to download multiple versions of the software.
4. Collect samples
=================================================
Your website samples should be representative of all SilverStripe installations. Take care not to just collect samples that are recently developed. Try to collect samples from a variety of sources and with a range of configurations.
Methods to find samples:
* Search Engines
* Website Showcases and Design Portfolios
* Forums for website development with the cms
Website Showcases
-----------------
A website showcase is a collection of websites that show off the abilities of the web designers and the potential of the CMS. Try to find showcases that have websites designed by more than one web developer. Sites that are made by the same developer are not properly representative of all sites and may include the designers idiosyncrasies.
While reading the background information I found this showcase on the official homepage: http://www.silverstripe.com/project-showcase/
By Googling for "silverstripe showcase" I found the official community showcase at http://www.silverstripe.org/community-showcase.
Googling for "webdesign portfolio silverstripe" found some web designers with links to SilverStripe websites.
The portfolio at http://smartplugsdesign.com/portfolio/ contains the following SilverStripe sites:
http://www.lisamarieelliott.com/
http://www.moonlitekustoms.com/
http://www.textiprints.com/
http://www.intandemtheatre.org/
http://www.stillrunnin.com/
http://www.enamaine.org/
The best source of samples is the community showcase because it contains a variety of websites made by different webdesigners and the websites are included in the portfolio over a period of time. Websites created over a wide period of time are useful as samples because they will run different versions of the SilverStripe software. There are 98 portfolio pages so I collected samples from pages 1, 25, 50, 75 and 98.
The samples collected from the community portfolio:
http://www.holistichealth.com/
http://www.verus.com.tr/
http://www.latenightdisco.com/
http://www.arprostatecancer.org/
http://www.cavendishimaging.com/
http://beatone.co.uk/
http://www.loguitos.com/
http://www.easycash4life.com/
http://www.gsbc.edu/
http://www.bradyinc.com/
http://www.monjasantner.de/
http://www.robert80.de/
http://customcanvas.fritzandandre.com/
http://www.idee-cruises.de/
http://www.maklerservice-greiz.de/
http://www.kitesurfnelson.co.nz/
http://www.moto-racepaint.com/
http://www.hutmacherin.com/
http://www.fuel.ie/silverstripe
http://www.infinitestillness.ie/ss
http://www.peterpanvakantieclub.nl/
http://www.chapmansurfboards.com/
http://www.fairtradenap.net/
http://www.benpearce.co.nz/
http://www.wend.nl/
http://www.resoba.com/
http://maungataniwha.co.nz/
http://www.gyo.co.nz/
http://www.firstgalaxies.org/
http://www.clockwork.co.nz/
http://www.upstreamgroup.com/
http://www.moerakihavenmotel.co.nz/
http://www.thelightboxdesigns.com/
http://www.nadabakery.co.nz/
http://comtel.com.au/
http://victoriaoruwari.com/
http://www.demconvention.com/
http://www.whileyouwait.co.nz/
http://omb.cl/
http://www.executivemediasearch.com/
http://www.naciondnb.com/
http://www.thecelebritytruth.com/
http://www.frussian.com.ar/
http://unbounded.org/
http://www.rcaforum.org.nz/
http://charcoalinteriors.com.au/
http://www.rcaforum.org.nz/
http://www.andrewking.co.nz/
http://www.elijahlofgren.com/silverstripe/
http://www.silverstripe.com/
This may seem like a large number of samples to collect but I assume that some of these websites will no longer be running SilverStripe or may no longer exist at all.
Using Search Engines
--------------------
Introduction
------------
Google-dorks are strings that can be used with Google to discover specific systems. There is an extensive database of google-dorks in the Google Hacking Database hosted at http://www.hackersforcharity.org/ghdb/.
Example: “Powered by Vsns Lemon†intitle:â€Vsns Lemonâ€
Using search engines to discover samples with google-dorks must not be the sole method used as these websites do not represent all sites on the internet running the system you are searching for. Webmasters have an incentive to remove the identifying strings discovered by google-dorks to reduce it's discoverability by malicious hackers.
Some WordPress installations include the text in the footer "Powered by WordPress" while this makes an excellent string to search for to find some installations, most WordPress sites do not include this string.
SilverStripe example
--------------------
First I searched for known google-dorks for SilverStripe by googling for "silverstripe google dork" and "silverstripe google hacking".
We won't know how to search for SilverStripe sites until we analyze some of the sample sites. Note that Google doesn't index html fragments, instead it just indexes words, titles, and urls.
I pick one sample to check, www.cavendishimaging.com. By reading the HTML source code I notice that the following line is included: <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
This positively identifies the site is made with SilverStripe but this content won't be indexed by google. At first glance nothing else on the page looks as though it can identify it was made with SilverStripe.
Forums for website development with the cms:
--------------------------------------------
Webdesign forums often have links to websites provided by the web designers. Some of these websites will be of lower quality than found in a portfolio and some will also be in the default setup. Such websites would not be included in an offical portfolio.
By Googling for "silverstripe webdesign forum" I found the official SilverStripe Forum: http://silverstripe.org/connect-with-other-silverstripe-members/show/256356
Some website samples collected from the forum are:
http://hungryhearts.no
http://weonline.in
http://belitsky.info/work/hartmann
http://kunstforum.as/
http://www.choidoco.com/demo/
http://www.tobychampion.co.uk/
http://www.silverstripe.org.pl/
5. Anaylze samples
=================================================
I need to analyze the samples I have collected to find similarities that can be used to identify these websites as SilverStripe. First I will search for identifying features in the webpages and HTTP headers.
In step 2 I collected 62 SilverStripe samples. I assume that some of these websites are incorrectly listed as SilverStripe so I will keep that in mind.
http://beatone.co.uk/
http://belitsky.info/work/hartmann
http://charcoalinteriors.com.au/
http://comtel.com.au/
http://customcanvas.fritzandandre.com/
http://hungryhearts.no
http://kunstforum.as/
http://maungataniwha.co.nz/
http://omb.cl/
http://unbounded.org/
http://victoriaoruwari.com/
http://weonline.in
http://www.andrewking.co.nz/
http://www.arprostatecancer.org/
http://www.benpearce.co.nz/
http://www.bradyinc.com/
http://www.cavendishimaging.com/
http://www.chapmansurfboards.com/
http://www.choidoco.com/demo/
http://www.clockwork.co.nz/
http://www.demconvention.com/
http://www.easycash4life.com/
http://www.elijahlofgren.com/silverstripe/
http://www.enamaine.org/
http://www.executivemediasearch.com/
http://www.fairtradenap.net/
http://www.firstgalaxies.org/
http://www.frussian.com.ar/
http://www.fuel.ie/silverstripe
http://www.gsbc.edu/
http://www.gyo.co.nz/
http://www.holistichealth.com/
http://www.hutmacherin.com/
http://www.idee-cruises.de/
http://www.infinitestillness.ie/ss
http://www.intandemtheatre.org/
http://www.kitesurfnelson.co.nz/
http://www.latenightdisco.com/
http://www.lisamarieelliott.com/
http://www.loguitos.com/
http://www.maklerservice-greiz.de/
http://www.moerakihavenmotel.co.nz/
http://www.monjasantner.de/
http://www.moonlitekustoms.com/
http://www.moto-racepaint.com/
http://www.naciondnb.com/
http://www.nadabakery.co.nz/
http://www.peterpanvakantieclub.nl/
http://www.rcaforum.org.nz/
http://www.resoba.com/
http://www.robert80.de/
http://www.silverstripe.com/
http://www.silverstripe.org.pl/
http://www.stillrunnin.com/
http://www.textiprints.com/
http://www.thecelebritytruth.com/
http://www.thelightboxdesigns.com/
http://www.tobychampion.co.uk/
http://www.upstreamgroup.com/
http://www.verus.com.tr/
http://www.wend.nl/
http://www.whileyouwait.co.nz/
Read the source of a couple of samples
--------------------------------------
Select at random 2 or 3 websites and read the HTML source carefully. Look for anything that isn't generic or anything that you wouldn't find on any website. Good places to scrutinise are headers, footers, url structures, filenames of javascript libraries and css files, and div naming schemes.
A fast visual inspection only identifies the meta generator tag:
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
A 2nd sample shows the following tag which includes a version number.
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
I also notice the URL format of some images is interesting, eg. "/assets/galleries/cakes/_resampled/Banner-Nada-090.jpg"
The div id names appear generic, eg. <div id="BgContainer">, <div id="Footer"> and <div class="footerTop">. At this stage they aren't interesting because I expect these div names to change with themes.
Collect HTML and HTTP headers from samples
------------------------------------------
Make a separate folder for the plugin you are analyzing. I will make the folder, plugin-development/tests/silverstripe/
$ cd whatweb-0.4/plugin-development/tests
$ mkdir silverstripe
$ cd silverstripe
Create a file in the silverstripe folder that contains the list of samples. I have called the file 'list'.
$ ../../wget-list
Usage: ../../wget-list <file with list of urls>
downloads each URL's html and headers into the current directory
In the plugin-development/ folder there is a script called wget-list. Use the script to download the samples into the silverstripe folder.
$ ../../wget-list ./list
--2010-03-04 17:03:09-- http://beatone.co.uk/
Resolving beatone.co.uk... 84.45.68.168
Connecting to beatone.co.uk|84.45.68.168|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `beatone.co.uk-.html'
[ <=> ] 10,785 19.2K/s in 0.5s
This takes a few minutes to complete. The script creates 2 files for each sample, an HTML and a META file which contains the HTTP headers.
$ ls
beatone.co.uk-.html www.demconvention.com-.meta www.moerakihavenmotel.co.nz-.meta
beatone.co.uk-.meta www.easycash4life.com-.html www.monjasantner.de-.html
belitsky.info-work-hartmann.html www.easycash4life.com-.meta www.monjasantner.de-.meta
belitsky.info-work-hartmann.meta www.elijahlofgren.com-silverstripe-.html www.moonlitekustoms.com-.html
charcoalinteriors.com.au-.html www.elijahlofgren.com-silverstripe-.meta www.moonlitekustoms.com-.meta
charcoalinteriors.com.au-.meta www.enamaine.org-.html www.moto-racepaint.com-.html
comtel.com.au-.html www.enamaine.org-.meta www.moto-racepaint.com-.meta
comtel.com.au-.meta www.executivemediasearch.com-.html www.naciondnb.com-.html
customcanvas.fritzandandre.com-.html www.executivemediasearch.com-.meta www.naciondnb.com-.meta
customcanvas.fritzandandre.com-.meta www.fairtradenap.net-.html www.nadabakery.co.nz-.html
hungryhearts.no.html www.fairtradenap.net-.meta www.nadabakery.co.nz-.meta
hungryhearts.no.meta www.firstgalaxies.org-.html www.peterpanvakantieclub.nl-.html
kunstforum.as-.html www.firstgalaxies.org-.meta www.peterpanvakantieclub.nl-.meta
kunstforum.as-.meta www.frussian.com.ar-.html www.rcaforum.org.nz-.html
list www.frussian.com.ar-.meta www.rcaforum.org.nz-.meta
maungataniwha.co.nz-.html www.fuel.ie-silverstripe.html www.resoba.com-.html
maungataniwha.co.nz-.meta www.fuel.ie-silverstripe.meta www.resoba.com-.meta
omb.cl-.html www.gsbc.edu-.html www.robert80.de-.html
omb.cl-.meta www.gsbc.edu-.meta www.robert80.de-.meta
unbounded.org-.html www.gyo.co.nz-.html www.silverstripe.com-.html
unbounded.org-.meta www.gyo.co.nz-.meta www.silverstripe.com-.meta
victoriaoruwari.com-.html www.holistichealth.com-.html www.silverstripe.org.pl-.html
victoriaoruwari.com-.meta www.holistichealth.com-.meta www.silverstripe.org.pl-.meta
weonline.in.html www.hutmacherin.com-.html www.stillrunnin.com-.html
weonline.in.meta www.hutmacherin.com-.meta www.stillrunnin.com-.meta
www.andrewking.co.nz-.html www.idee-cruises.de-.html www.textiprints.com-.html
www.andrewking.co.nz-.meta www.idee-cruises.de-.meta www.textiprints.com-.meta
www.arprostatecancer.org-.html www.infinitestillness.ie-ss.html www.thecelebritytruth.com-.html
www.arprostatecancer.org-.meta www.infinitestillness.ie-ss.meta www.thecelebritytruth.com-.meta
www.benpearce.co.nz-.html www.intandemtheatre.org-.html www.thelightboxdesigns.com-.html
www.benpearce.co.nz-.meta www.intandemtheatre.org-.meta www.thelightboxdesigns.com-.meta
www.bradyinc.com-.html www.kitesurfnelson.co.nz-.html www.tobychampion.co.uk-.html
www.bradyinc.com-.meta www.kitesurfnelson.co.nz-.meta www.tobychampion.co.uk-.meta
www.cavendishimaging.com-.html www.latenightdisco.com-.html www.upstreamgroup.com-.html
www.cavendishimaging.com-.meta www.latenightdisco.com-.meta www.upstreamgroup.com-.meta
www.chapmansurfboards.com-.html www.lisamarieelliott.com-.html www.verus.com.tr-.html
www.chapmansurfboards.com-.meta www.lisamarieelliott.com-.meta www.verus.com.tr-.meta
www.choidoco.com-demo-.html www.loguitos.com-.html www.wend.nl-.html
www.choidoco.com-demo-.meta www.loguitos.com-.meta www.wend.nl-.meta
www.clockwork.co.nz-.html www.maklerservice-greiz.de-.html www.whileyouwait.co.nz-.html
www.clockwork.co.nz-.meta www.maklerservice-greiz.de-.meta www.whileyouwait.co.nz-.meta
www.demconvention.com-.html www.moerakihavenmotel.co.nz-.html
The folder whatweb-0.4/plugin-development/tests/silverstripe now contains many .html and .meta files.
$ head beatone.co.uk-.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<base href="http://beatone.co.uk/" ><!--[if IE 6]></base><![endif]-->
<title>Be At One - London Bar, Bookings Central London, Great Cocktails London </title>
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
<meta http-equiv="Content-type" content="text/html; charset=utf-8" >
<meta http-equiv="Content-Language" content="en-US">
<link rel="shortcut icon" href="/favicon.ico">
This is a standard HTML file, this is the same as what you see when you select 'View Source' in a web browser.
$ cat beatone.co.uk-.meta
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:35 GMT
Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8+etch16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,User-Agent,Accept-Encoding
Content-Type: text/html; charset=utf-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4d463f54abb74031c117569ca3aa3c61; path=/
These are the HTTP Headers the webserver sends before the HTML. Look for unusual cookie names and non-standard HTTP headers.
Remove incorrectly identified samples
-------------------------------------
To identify some samples that are not SilverStripe I grep for the generator tag and remove all occurances that include the word Silver. This leaves me with the following:
$ grep generator *html | grep -v Silver
omb.cl-.html: <meta name="generator" content="dospuntocero.cl" >
www.andrewking.co.nz-.html:<meta name="generator" content="WordPress 2.8.4" />
www.easycash4life.com-.html:<meta name="generator" content="WordPress 2.8.2" />
www.idee-cruises.de-.html:<meta name="generator" http-equiv="generator" content="cms.Koncepts - http://www.koncepts.de" />
www.thecelebritytruth.com-.html:<meta name="generator" content="WordPress 2.8.4" />
These websites are obviously not SilverStripe so I delete their files and remove them from the list.
$ rm omb.cl-.* www.andrewking.co.nz-.* www.easycash4life.com-.* www.idee-cruises.de-.* www.thecelebrity truth.com-.*
How many sites can we be certain are SilverStripe?
--------------------------------------------------
There are 57 website samples left:
$ ls *html | wc -l
57
Of these samples, 38 include the term silverstrip in the HTML
$ grep -li silverstripe *html | wc -l
38
Examine the samples with WhatWeb
--------------------------------
Using whatweb before the plugin is written may show some interesting information. In this case it has identified the meta generator tag. However notice that some of these websites do not have the meta generator tag. Reasons could be that the webmaster has removed it or the website is no longer running SilverStripe.
This is also useful to find more samples that are not SilverStripe
$ ./whatweb -i ./plugin-development/tests/silverstripe/list
ttp://belitsky.info/work/hartmann [301] md5[c112335e6a56038ca4ba4b906d6aee05], redirect-location[http://belitsky.info/work/hartmann/], server-header[Apache], title[301 Moved Permanently]
http://belitsky.info/work/hartmann/ [200] index-of, md5[21577203b9abc6091d99203295712f0c], server-header[Apache], title[Index of /work/hartmann]
http://charcoalinteriors.com.au/ [200] md5[bff9a28ebdc1cdfdb80743458606df1d], server-header[Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Home -Charcoal Interiors], x-powered-by-header[PHP/5.2.10]
http://customcanvas.fritzandandre.com/ [200] JQuery, Mailto, md5[452d4fd540f07c98e0288094d0bf959f], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ruby/1.2.6 Ruby/1.8.7(2008-08-11) mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Home], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://comtel.com.au/ [200] Google-Analytics-GA[1388941], probably Joomla[com_search], md5[050d35f63e2d9cd46065cde83f876989], server-header[Apache/2.0.55 (Ubuntu) PHP/5.1.2], title[Comtel - Telephone Radio & Data Systems | Comtel], x-powered-by-header[PHP/5.1.2]
http://beatone.co.uk/ [200] Google-Analytics-GA[11953167], JQuery, md5[b85a0567c28cfc3ba050ccbc95c899c4], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Be At One - London Bar, Bookings Central London, Great Cocktails London], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://kunstforum.as/ ERROR: Socket error getaddrinfo: Name or service not known
http://hungryhearts.no [200] Google-Analytics-GA[2984373], JQuery, Mailto, md5[1156688f57dfb37d853d3d7326daaadc], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_psoft_traffic/0.2 mod_ssl/2.8.31 OpenSSL/0.9.7a mod_macro/1.1.2], title[The Hungry Hearts. Pin-up performance band.], x-powered-by-header[PHP/5.2.6]
http://unbounded.org/ [200] Google-Analytics-urchin[97930], md5[34d6c3cfc3b9ba0c66a8942a490c259d], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2.2.3 (Debian) DAV/2 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_ssl/2.2.3 OpenSSL/0.9.8g], title[unbounded], x-powered-by-header[PHP/5.2.6-1+lenny3]
http://www.arprostatecancer.org/ [200] Google-Analytics-GA[2447233], Mailto, md5[dc1a9f02efc52b1ebc72f2c8a0b03ae6], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Arkansas Prostate Cancer Foundation], x-powered-by-header[PHP/5.2.6]
http://weonline.in [200] Google-Analytics-GA[8297705], Mailto, md5[f9d784e8b18c942fe7ea4ed8273630c9], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache], title[Home. Weonline web design group. We love to do beautiful stuff for the web.], x-powered-by-header[PHP/5.2.9]
http://maungataniwha.co.nz/ [200] Google-Analytics-GA[3842018], JQuery, md5[58fbcc50e566ceaab813ecd38098e1df], server-header[Apache/2.2], title[Maungataniwha Lodge | New Zealand | Home]
http://victoriaoruwari.com/ [200] md5[2afe04307f5e3503efbe1c2c75b62511], server-header[Apache/1.3.41 (Unix) mod_ssl/2.8.31 OpenSSL/0.9.7a PHP/5.2.8 mod_perl/1.29 FrontPage/5.0.2.2510], title[Victoria Oruwari - Home], x-powered-by-header[PHP/5.2.8]
http://www.benpearce.co.nz/ [200] Google-Analytics-GA[1362535], JQuery, Mailto, Prototype, md5[4b1abe022c1c40c6091366c71c367cca], server-header[Apache/2.0.54 (Debian GNU/Linux) PHP/5.2.3-0.dotdeb.0 with Suhosin-Patch mod_ssl/2.0.54 OpenSSL/0.9.7e], title[Ben Pearce - artist], x-powered-by-header[PHP/5.2.3-0.dotdeb.0]
http://www.chapmansurfboards.com/ [200] Google-Analytics-GA[419314], JQuery, md5[9036620b334c3d2e7f6722e946277d29], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache], title[Dale Chapman Surf Designs], x-powered-by-header[PHP/5.2.10]
http://www.cavendishimaging.com/ [200] Google-Analytics-GA[11469477], JQuery, md5[ef8fbfae1dec5750a55478ed295cb34d], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Dentomaxillofacial Imaging & Anatomical Model Specialists - Cavendish Imaging], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://www.demconvention.com/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.clockwork.co.nz/ [200] md5[70e4b614d4ae85161014d05939ebd073], server-header[Apache], title[clockwork.co.nz]
http://www.choidoco.com/demo/ [200] md5[ccfe1940e9651416af58ff3f4a3eff77], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.11 mod_perl/2.0.4 Perl/v5.8.8], title[home], x-powered-by-header[PHP/5.2.11]
http://www.bradyinc.com/ [200] Google-Analytics-GA[13121212], Prototype, md5[34d5c1f763e3ceffaeae07de7de98f3d], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7m], title[Staffing Productivity Benchmarks » Brady & Associates], x-powered-by-header[PHP/5.2.11]
http://www.executivemediasearch.com/ [404] md5[588da43361637cd97f3096ab9ce70183], server-header[Apache], title[Error 404 - Not found]
http://www.fairtradenap.net/ [200] Google-Analytics-GA[1362535], Mailto, md5[f4f36e648290b8fb818ea7fb297a4944], server-header[Apache], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.elijahlofgren.com/silverstripe/ [404] Google-Analytics-urchin[2328965], maybe Mambo, md5[89a0d093054c83ef60a842b2aa7ff48f], meta-generator[CMS Made Simple - Copyright (C) 2004-6 Ted Kulp. All rights reserved.], powered by...[CMSMS], server-header[lighttpd/1.4.22], title[404 Error - Elijah Lofgren's Website]
http://www.enamaine.org/ [200] Google-Analytics-GA[3359251], Mailto, md5[57cb1e27c565acff11ce5f8103696696], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Maine ENA Home | Maine ENA], x-powered-by-header[PHP/5.2.9]
http://www.fuel.ie/silverstripe [301] md5[cbee7d5cfda4e161caffa892cc08558a], redirect-location[http://www.fuel.ie/silverstripe/], server-header[Zeus/4.3], title[Error 301 Moved Permanently]
http://www.firstgalaxies.org/ [200] Google-Analytics-urchin[777185], md5[fde922a270e0b438789eef03f2bbc064], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[A Resource for Research on the Most Distant Galaxies], x-powered-by-header[PHP/5.2.11]
http://www.frussian.com.ar/ [200] md5[54b8d6e69f5be47d29f8225126bf92da], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.gyo.co.nz/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.gsbc.edu/ [200] Google-Analytics-GA[276990], JQuery, Prototype, md5[dc166f92e1ed43f5435f60743c0d272a], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635], title[Home » Golden State Baptist College], x-powered-by-header[PHP/5.2.11]
http://www.fuel.ie/silverstripe/ [200] Mailto, md5[c63c8ea00aa0624fb4df7989c92b172e], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Zeus/4.3], title[The Fuel/Silverstripe Demo Site » The Fuel/Silverstripe Demo Site]
http://www.infinitestillness.ie/ss [301] md5[cbee7d5cfda4e161caffa892cc08558a], redirect-location[http://www.infinitestillness.ie/ss/], server-header[Zeus/4.3], title[Error 301 Moved Permanently]
http://www.holistichealth.com/ [200] Google-Analytics-GA[6289330], md5[b59e703ec114ea8cd99da42af210f1a1], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Holistic Health International - Where Science and Caring Meet], x-powered-by-header[PHP/5.2.6]
http://www.hutmacherin.com/ [301] md5[d41d8cd98f00b204e9800998ecf8427e], redirect-location[/start], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g]
http://www.infinitestillness.ie/ss/ [200] Mailto, md5[77b579a3d6752d45bcd8718d906fe5c3], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Zeus/4.3], title[Infinite Stillness | Ki Massage & Reiki Healing Dublin 4]
http://www.hutmacherin.com/start [200] md5[a89fbf86cf798314dba620717b1d99b8], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Start | Isabell von Maltzahn | Hutmacherin aus Berlin]
http://www.intandemtheatre.org/ [200] Google-Analytics-GA[6603467], Prototype, md5[133a6893f85c4c41034ced6a7aec3e75], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Welcome | In Tandem Theatre], x-powered-by-header[PHP/5.2.9]
http://www.latenightdisco.com/ [200] Google-Analytics-GA[768894], md5[b81b6b18af21e3d5f7d0749aa483da64], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch], title[Experience Central Arkansas hottest night club Discovery], x-powered-by-header[PHP/5.2.4-2ubuntu5.4]
http://www.lisamarieelliott.com/ [200] Google-Analytics-GA[3359251], md5[d844eb0937b79fc02c347930522fe490], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.loguitos.com/ [200] Joomla[1.0], maybe Mambo, md5[0eed48ee56f6a2b784ea010b1bfa15b8], meta-generator[Joomla! - Copyright (C) 2005 - 2007 Open Source Matters. All rights reserved.], server-header[Apache/2.2.10 (Unix) mod_ssl/2.2.10 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Dise�o de logotipos - Loguitos - Dise�adores de logos profesionales - Inicio], x-powered-by-header[PHP/5.2.6]
http://www.moonlitekustoms.com/ [200] md5[cf3d07ae32e645d04a1acad6560ce668], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[In The Shop - Moonlite Kustoms], x-powered-by-header[PHP/5.2.9]
http://www.maklerservice-greiz.de/ [200] Google-Analytics-GA[10587433], Prototype, md5[9ac4c5710cf3e694917e2a3949680fc7], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.2635], title[Ihr Maklerservice in Greiz: Steiniger Versicherungsmakler], x-powered-by-header[PHP/5.2.9]
http://www.moerakihavenmotel.co.nz/ [200] md5[12e0be3129dea7738500b21aeab0ff96], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], powered by...[:], server-header[Apache], title[Moeraki Haven Motel, Moreaki Motel Accommodation, Otago Motel Accommodation.], x-powered-by-header[PHP/5.2.11]
http://www.monjasantner.de/ [200] Google-Analytics-GA[10599117], JQuery, md5[c30a4c5e6b9aad2a7f435621deb2ef40], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Monja Santner » Home], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://www.nadabakery.co.nz/ [200] Google-Analytics-GA[4761582], md5[68f3dd581e40f370927205103ca6903a], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache/2], title[Home | Nada - New Zealand's Greatest Bakery], x-powered-by-header[PHP/5.2.12]
http://www.naciondnb.com/ [200] md5[d443c89dc137c8286aaa173ebc806176], server-header[Apache], title[NacionDNB]
http://www.moto-racepaint.com/ [200] Google-Analytics-GA[1912569], JQuery, md5[7c1db45c5801d09dac4539725c6cf658], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4], title[MotoRacePaint Home], x-powered-by-header[PHP/5.2.11]
http://www.rcaforum.org.nz/ [200] Google-Analytics-GA[4693659], Mailto, Prototype, md5[5cbb333d29fb180f42142a9551e105a2], meta-generator[SilverStripe 2.3.0 - http://www.silverstripe.com], server-header[Apache], title[Homepage - RCA Forum], x-powered-by-header[PHP/5.2.1]
http://www.peterpanvakantieclub.nl/ [200] Google-Analytics-GA[1010482], JQuery, md5[011e5aa0e0e9599d10cd2913270959ea], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2], title[Peter Pan Vakantieclub | Home], x-powered-by-header[PHP/5.2.4]
http://www.robert80.de/ [200] Lightbox, md5[e12ecd879e7a24eaf6630d0097f6d0c8], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Robert Müller 80 Freunde » Robert Müller 80 Freunde helfen], x-powered-by-header[PHP/5.2.13]
http://www.silverstripe.com/ [200] Google-Analytics-urchin[84547], JQuery, md5[25a7a15b8e40a9443ed6bd896705b7ba], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.11 (Debian) PHP/5.2.6-1+lenny2 with Suhosin-Patch mod_ssl/2.2.11 OpenSSL/0.9.8g], title[SilverStripe.com - Open Source CMS / Framework], x-powered-by-header[PHP/5.2.6-1+lenny2]
http://www.stillrunnin.com/ [200] Google-Analytics-GA[3359251], md5[bd3f631ef6075169b66c44031e526184], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Still Runnin Magazine - Online Gearhead Ezine], x-powered-by-header[PHP/5.2.9]
http://www.silverstripe.org.pl/ [200] Google-Analytics-GA[8121843], md5[1d8badb4df0d05a72c665de18087ae2b], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.8], title[Serwis polskiej spoÅ‚ecznoÅ›ci SilverStripe » SilverStripe.org.pl], x-powered-by-header[PHP/5.2.8]
http://www.thelightboxdesigns.com/ [200] Mailto, md5[763842fb73491b1ccc9090219e299ed1], server-header[Apache], title[Graphic and Web Site Design Services in Brownsville, TX, McAllen, Harlingen and the Rio Grande Valley :: The Lightbox Designs]
http://www.textiprints.com/ [200] Google-Analytics-GA[10064088], md5[65c429cfbad7a5acccddb8eda664f13c], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[TextiPrints - Digital Garment Printer - Ormond Beach, Florida], x-powered-by-header[PHP/5.2.9]
http://www.tobychampion.co.uk/ [500] md5[c5ea88dc871f71751f932a8a9bed884b], server-header[Apache/2.0.63 (FreeBSD) DAV/2 SVN/1.5.2 mod_python/3.3.1 Python/2.5.1 PHP/5.2.6 with Suhosin-Patch mod_ssl/2.0.63 OpenSSL/0.9.7e-p1 mod_fastcgi/2.4.6 mod_perl/2.0.3 Perl/v5.8.8], title[GET /], x-powered-by-header[PHP/5.2.12]
http://www.upstreamgroup.com/ [200] Google-Analytics-GA[3522744], md5[085964a490aa87a8d061828ebd63f35b], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache], title[Upstream Group: Clarity, Perspective, Knowledge], x-powered-by-header[PHP/5.2.11]
http://www.wend.nl/ [200] Google-Analytics-GA[1010482], JQuery, md5[e08dcd1b16e7210f1762b49bb45d58ff], server-header[Apache/2], title[Wend - Home], x-powered-by-header[PHP/5.2.4]
http://www.verus.com.tr/ [200] Google-Analytics-GA[7233761], md5[54bfe62b41b88b472d9739d2ed47b30f], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8], title[VERUS » ETKÄ°NLÄ°K ÇÖZÃœMLERÄ° » Ä°Åž ÇÖZÃœMLERÄ° » WEB ÇÖZÃœMLERÄ°], x-powered-by-header[PHP/5.2.4-2ubuntu5.10]
http://www.whileyouwait.co.nz/ [200] Mailto, md5[05df26cf45f8410dea2272f6c0ff269b], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache], title[While You Wait Studios - Christchurch, New Zealand - PTFOTO], x-powered-by-header[PHP/5.2.11]
http://www.kitesurfnelson.co.nz/ [200] Google-Analytics-GA[1921819], md5[f7618ec4d8da26579e5df948d80ba5b8], server-header[Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Kite Surf Nelson - kitesurfing lessons, equipment sales and advice - in Nelson, New Zealand], x-powered-by-header[PHP/5.2.6]
http://www.resoba.com/ ERROR: EOF error end of file reached
Remove more incorrectly identified samples with the whatweb report
------------------------------------------------------------------
http://belitsky.info/work/hartmann [301] md5[c112335e6a56038ca4ba4b906d6aee05], redirect-location[http://belitsky.info/work/hartmann/], server-header[Apache], title[301 Moved Permanently]
http://belitsky.info/work/hartmann/ [200] index-of, md5[21577203b9abc6091d99203295712f0c], server-header[Apache], title[Index of /work/hartmann]
This matches the 'Index of' plugin. By loading this URL into a web browser I can see that this webpage isn't a CMS, instead it's a directory listing so I can delete http://belitsky.info/work/hartmann from the list.
http://charcoalinteriors.com.au/ [200] md5[bff9a28ebdc1cdfdb80743458606df1d], server-header[Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Home -Charcoal Interiors], x-powered-by-header[PHP/5.2.10]
There's no way to be sure if this is SilverStripe because it has the meta generator tag removed.
http://comtel.com.au/ [200] Google-Analytics-GA[1388941], probably Joomla[com_search], md5[050d35f63e2d9cd46065cde83f876989], server-header[Apache/2.0.55 (Ubuntu) PHP/5.1.2], title[Comtel - Telephone Radio & Data Systems | Comtel], x-powered-by-header[PHP/5.1.2]
This appears to be powered by the Joomla CMS. A website cannot be two CMSs at the same time with the same URL so if Joomla is present then SilverStripe cannot be. Note that other plugin matches such as Jquery identify a javascript library and will be found with many different CMSs including SilverStripe. Manual verification by testing http://comtel.com.au/administrator proves it is Joomla.
http://kunstforum.as/ ERROR: Socket error getaddrinfo: Name or service not known
This website doesn't exist anymore.
http://www.arprostatecancer.org/ [200] Google-Analytics-GA[2447233], Mailto, md5[dc1a9f02efc52b1ebc72f2c8a0b03ae6], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Arkansas Prostate Cancer Foundation], x-powered-by-header[PHP/5.2.6]
I can't be sure if this is SilverStripe yet.
I deleted the files of the remaining websites that are identified as something other than SilverStream.
$ wc -l list
49 list
Now I have just 49 SilverStripe samples. I know that some of these samples might not be SilverStripe.
Use find-common-stuff to automatically identify common strings in the samples
-----------------------------------------------------------------------------
I can use a simple tool called find-common-stuff to find certain types of common strings in samples.
find-common-stuff will identify and count the occurances of:
* complete HTML tags
* strings enclosed in double quotes
It has threshold setting that adjusts how many uncommon things are displayed.
$ ../../find-common-stuff
Usage: find-common-stuff FILES
--threshold, -t The lowest % of files an item occurs in to display. Eg. 0.25 and 0.50
$ ../../find-common-stuff *html
imported 49 files
counted 3324 tags
[["<script type=\"text/javascript\">", 35],
["<![endif]-->", 30],
["<meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />",
28],
["<style type=\"text/css\">", 21],
["<!--[if IE 7]>", 21],
["<!--[if IE 6]>", 21],
["<meta name=\"generator\" http-equiv=\"generator\" content=\"SilverStripe - http://www.silverstripe.com\" />",
20],
["<div id=\"footer\">", 16],
["<link rel=\"shortcut icon\" href=\"/favicon.ico\" />", 16],
["<div class=\"clear\">", 16],
["<meta http-equiv=\"Content-Language\" content=\"en-US\"/>", 14],
["<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.1//EN\" \"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\">",
14],
["<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">",
14],
["<div class=\"typography\">", 14],
["<meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" >",
13]]
counted 1874 quoted texts
[["\"text/javascript\"", 35],
["\"link\"", 24],
["\"text/css\"", 23],
["\"typography\"", 19],
["\"current\"", 19],
["\"clear\"", 18],
["\"footer\"", 17],
["\"en\"", 16],
["\"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\"", 14],
["\"http://www.w3.org/TR/html4/strict.dtd\"", 14],
["\"header\"", 14]]
In this case, the automated tool, find-common-stuff has failed to identify anything I didn't noticed while reading the HTML source.
Analyse HTTP headers and cookies
--------------------------------
The .meta files contain the HTTP headers and any cookies set by the websites.
Use the cat command to display all of them.
$ cat *meta
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:35 GMT
Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8+etch16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,User-Agent,Accept-Encoding
Content-Type: text/html; charset=utf-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4d463f54abb74031c117569ca3aa3c61; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:41 GMT
Server: Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.10
Content-Type: text/html
Via: 1.1 bc6
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:30 GMT
Server: Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ruby/1.2.6 Ruby/1.8.7(2008-08-11) mod_ssl/2.2.9 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=1b627d14a21e4475c49e7089c01e11b8; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:49 GMT
Server: Apache/1.3.41 (Unix) PHP/5.2.6 mod_psoft_traffic/0.2 mod_ssl/2.8.31 OpenSSL/0.9.7a mod_macro/1.1.2
X-Powered-By: PHP/5.2.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=kibbf9utq9decif304ml5lgu01; path=/
HTTP/1.1 200 OK
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Date: Thu, 04 Mar 2010 04:05:09 GMT
Server: Apache/2.2
Content-Type: text/html; charset="utf-8"
Pragma: no-cache
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=tmufihalkg02p4t0vi3mdf3k94; path=/
Set-Cookie: X-Mapping-caklakng=293486F930CB5202C46B0E5EFB41C64E; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:29 GMT
Server: Apache/2.2.3 (Debian) DAV/2 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_ssl/2.2.3 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny3
Set-Cookie: PHPSESSID=1b0b76cadaefb10b5973b8bb622d9669; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
Vary: Accept,Accept-Encoding
Content-Type: text/html; charset=utf-8
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:35 GMT
Server: Apache/1.3.41 (Unix) mod_ssl/2.8.31 OpenSSL/0.9.7a PHP/5.2.8 mod_perl/1.29 FrontPage/5.0.2.2510
X-Powered-By: PHP/5.2.8
Expires: Mon, 21 Jun 2010 11:11:02 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Sat, 14 Nov 2009 21:00:08 GMT
Vary: Accept
Content-Type: text/html; charset=utf-8
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=b255ace17a48eadebd5de6ec6b7f3dc6; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:37 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5nfa7b62smslu5qo660bmtmk32; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:34 GMT
Server: Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a
X-Powered-By: PHP/5.2.6
Content-Type: text/html
Via: 1.1 bc7
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:46 GMT
Server: Apache/2.0.54 (Debian GNU/Linux) PHP/5.2.3-0.dotdeb.0 with Suhosin-Patch mod_ssl/2.0.54 OpenSSL/0.9.7e
X-Powered-By: PHP/5.2.3-0.dotdeb.0
Set-Cookie: PHPSESSID=d627a8e8613eecd6b0c2a41b0ec79dd4; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:50 GMT
Server: Apache/1.3.41 (Unix) FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7m
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.11
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=9508d9fa7a9d869e745c286ff9799d40; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:54 GMT
Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8+etch16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,User-Agent,Accept-Encoding
Content-Type: text/html; charset=utf-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=8d9b19f46268342f8a0823bd79599cd2; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:01 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Expires: Sat, 13 Mar 2010 04:54:33 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Tue, 23 Feb 2010 03:17:29 GMT
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4askloa60f26kvftt1cfiao344; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:13 GMT
Server: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.11 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform
Pragma: no-cache
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
X-Cache: MISS from sv25.byethost25.org
Via: 1.0 sv25.byethost25.org:80 (squid/2.7.STABLE7), 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f0c426eaeef84f38c04624a59a98edd4; path=/demo/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:17 GMT
Server: Apache
Expires: Thu, 29 Oct 1998 17:04:19 GMT
Last-Modified: Thu, 04 Mar 2010 04:06:17 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: clockwork_co_nz=4040a20413b27678f6a5b225bd85f612; expires=Tue, 03-Mar-2015 04:06:17 GMT; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:46 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=6e9a4bb3b41cdd968202a19fbb91b3c0; path=/
HTTP/1.1 404 Not Found
Date: Thu, 04 Mar 2010 04:06:48 GMT
Server: Apache
Content-Type: text/html
Via: 1.1 bc1
Content-Length: 0
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:49 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=ec0a2031ab05b371baf412429859bbdf; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:01 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Vary: Accept,Accept-Encoding
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset=utf-8
Via: 1.1 bc4
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5e61do18cag74urkrq65eqkcf3; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:07 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.9
Cache-Control: max-age=86400, must-revalidate
Pragma:
Expires: Wed, 14 Mar 2012 07:10:47 GMT
Vary: Accept
Last-Modified: Fri, 22 Feb 2008 01:03:27 GMT
Content-Type: text/html; charset=utf-8
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=3fb099aeb12584c9a2a308667e960553; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/
HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Mar 2010 04:07:14 GMT
Location: http://www.fuel.ie/silverstripe/
Server: Zeus/4.3
Content-Type: text/html
Via: 1.1 bc3
Content-Length: 212
Connection: Keep-Alive
Set-Cookie: X-Mapping-enlokcai=E05A570E7E395D6AB72BEA6FC2D4D8D4; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:19 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=05b506d2ba68f9057269c4ff70ff9643; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:11 GMT
Server: Apache/1.3.41 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.6
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f7aec63e8640dcd5d42a0476301a065f; path=/
HTTP/1.1 301 OK
Date: Thu, 04 Mar 2010 04:07:27 GMT
Server: Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: /start
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=0e75b5e83a844f8902e361394c39c39f; path=/
HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Mar 2010 04:07:36 GMT
Location: http://www.infinitestillness.ie/ss/
Server: Zeus/4.3
Content-Type: text/html
Via: 1.1 bc3
Content-Length: 212
Connection: Keep-Alive
Set-Cookie: X-Mapping-enlokcai=E05A570E7E395D6AB72BEA6FC2D4D8D4; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:48 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=75cd9c74439efc3bca8a6f6a8fd429f4; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:48 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=e5fa1ae0a7149cb25f36f4cf63e55603; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:54 GMT
Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch
X-Powered-By: PHP/5.2.4-2ubuntu5.4
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4c256b274d52c2746423786faadd30c4; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:00 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=bb34c4a17da632e7aefab6246e7704ab; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:06 GMT
Server: Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.2635
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.9
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=416bd54449181a0b0839ae7dcda95902; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:06 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Cache-Control: max-age=86400, must-revalidate
Pragma:
Expires: Sat, 09 Oct 2010 04:35:26 GMT
Vary: Accept
Set-Cookie: PHPSESSID=c977eedf73ae811cabbe26bc8e0313c5; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
Last-Modified: Tue, 28 Jul 2009 03:40:46 GMT
Content-Type: text/html; charset=utf-8
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:11 GMT
Server: Apache
X-Powered-By: PHP/5.2.6-1+lenny6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=e0a6ca2c8e29af432c8a1266e992e129; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:17 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=0423b5407d22301ff1e4b3d2f41b9970; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:14 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=6ad778ad438fca511d10c7f438d181dd; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:20 GMT
Server: Apache
Last-Modified: Mon, 17 Aug 2009 04:07:41 GMT
ETag: "7c8c026-3aa-8b0b3d40"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Type: text/html
Via: 1.1 bc1
Content-Length: 938
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:20 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.12
Set-Cookie: PHPSESSID=78e42396dc632a12633ef0f065b979be; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:09:47 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.4
Expires: Fri, 19 Mar 2010 21:22:18 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Tue, 16 Feb 2010 10:57:16 GMT
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=2055abb25c60b3a28fc9810dd601fe13; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:24 GMT
Server: Apache
X-Powered-By: PHP/5.2.1
Set-Cookie: PHPSESSID=b6cc6c9f2dc42acb0dca07bd778b031c; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:26 GMT
Server: Apache
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.13
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=40555488da9765a46c63a96625ad28b6; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:27 GMT
Server: Apache/2.2.11 (Debian) PHP/5.2.6-1+lenny2 with Suhosin-Patch mod_ssl/2.2.11 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny2
Set-Cookie: PHPSESSID=1f41c7f80dfad6ef9e9594963612472f; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,Accept-Encoding
Content-Type: text/html; charset=utf-8
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:13 GMT
Server: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.8
X-Powered-By: PHP/5.2.8
Content-Type: text/html
Via: 1.1 bc5
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:35 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f0ca06da67663004b31d8f14f721daab; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:40 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=98f72bf1ed2792b143f47308b3b5f099; path=/
HTTP/1.1 403 Forbidden
Date: Thu, 04 Mar 2010 04:28:42 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 bc5
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:45 GMT
Server: Apache/2.0.63 (FreeBSD) DAV/2 SVN/1.5.2 mod_python/3.3.1 Python/2.5.1 PHP/5.2.6 with Suhosin-Patch mod_ssl/2.0.63 OpenSSL/0.9.7e-p1 mod_fastcgi/2.4.6 mod_perl/2.0.3 Perl/v5.8.8
X-Powered-By: PHP/5.2.12
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=860147a02be38690829b458077eabd7c; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:47 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=t4962e1dunk93crufg8qil4375; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:53 GMT
Server: Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8
X-Powered-By: PHP/5.2.4-2ubuntu5.10
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5a5baaf3c09bd237859f534039e52f03; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:30:20 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.4
Expires: Tue, 08 Jun 2010 22:50:34 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Fri, 27 Nov 2009 10:10:06 GMT
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=2089013901ea310463177f049f4a92b4; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:55 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept
Set-Cookie: PHPSESSID=b63d92df808dc2f1edafa1c1e8830409; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/
Content-Type: text/html; charset=utf-8
I noticed two uncommon HTTP headers. The expiry date in 1981 and the cookie, PastVisitor.
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/
Googling for 'cookie "PastVisitor"' turns up results referring directly to SilverStripe and results referring to websites running SilverStripe. This cookie name while generic sounding appears to be only used by SilverStripe and makes a good plugin pattern.
Googling for 'Thu, 19 Nov 1981 08:52:00 GMT' turns up many results. This date relates to the PHP language and is not useful in identifying SilverStripe.
Read more HTML source
---------------------
Select some more HTML files to read, looking for unusual patterns.
grep all the HTML files simulatenously looking for patterns:
Examples:
$ grep css *html
$ grep javascript *html
Many of the samples have a css file called typography.css. This by itself isn't uncommon enough to make a plugin match. Even if we search for themes/.*/css/typography.css it's still not uncommon enough.
<link rel="stylesheet" type="text/css" href="http://www.lisamarieelliott.com/themes/lisamarieelliott/css/typography.css?m=1254246770" />
The -A parameter to the grep command is used to display lines after the matched line. Using this we can see the lines directly after layout.css.
$ grep -A 2 layout.css *html
customcanvas.fritzandandre.com-.html:<link rel="stylesheet" type="text/css" href="http://customcanvas.fritzandandre.com/themes/blueplanet/css/layout.css?m=1254524509" />
customcanvas.fritzandandre.com-.html-<link rel="stylesheet" type="text/css" href="http://customcanvas.fritzandandre.com/themes/blueplanet/css/typography.css?m=1254524509" />
customcanvas.fritzandandre.com-.html-<link rel="stylesheet" type="text/css" href="http://customcanvas.fritzandandre.com/themes/blueplanet/css/form.css?m=1254524509" />
--
hungryhearts.no.html: <link rel="stylesheet" href="themes/hh/css/layout.css" type="text/css">
hungryhearts.no.html- <link rel="stylesheet" href="themes/hh/css/form.css" type="text/css">
hungryhearts.no.html- <link rel="stylesheet" href="themes/hh/javascript/fancybox/jquery.fancybox.css" type="text/css" media="screen">
--
maungataniwha.co.nz-.html:<link rel="stylesheet" type="text/css" href="http://maungataniwha.co.nz/themes/maungataniwha/css/layout.css?m=1265149666" />
maungataniwha.co.nz-.html-<link rel="stylesheet" type="text/css" href="http://maungataniwha.co.nz/themes/maungataniwha/css/typography.css?m=1265149666" />
maungataniwha.co.nz-.html-<link rel="stylesheet" type="text/css" href="http://maungataniwha.co.nz/themes/maungataniwha/css/form.css?m=1265149668" />
--
weonline.in.html: <link rel="stylesheet" href="themes/weonline/css/layout.css" type="text/css" media="screen" />
weonline.in.html-
weonline.in.html- <!--[if IE 6]>
--
www.benpearce.co.nz-.html: <link href="themes/main/css/layout.css" rel="stylesheet" type="text/css" />
www.benpearce.co.nz-.html- <link href="themes/main/css/typography.css" rel="stylesheet" type="text/css" />
www.benpearce.co.nz-.html- <link href="themes/main/css/form.css" rel="stylesheet" type="text/css" />
--
www.bradyinc.com-.html: <link rel="stylesheet" type="text/css" href="http://www.bradyinc.com/themes/bradyassociates/css/layout.css?m=1267052060" />
www.bradyinc.com-.html-<link rel="stylesheet" type="text/css" href="http://www.bradyinc.com/themes/bradyassociates/css/typography.css?m=1266644617" />
www.bradyinc.com-.html-<link rel="stylesheet" type="text/css" href="http://www.bradyinc.com/themes/bradyassociates/css/form.css?m=1266644611" />
--
layout.css itself is not uncommon enough to make a plugin match. However many of the samples have at least 3 css files named layout.css, typography.css and form.css. The use of these names is not exclusive to SilverStripe and is considered best practice for making CSS frameworks but the order of their appearance combined with the folder structure is unique enough for a 'probable' plugin match.
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/layout.css?m=1266347738" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/typography.css?m=1266347623" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/form.css?m=1247030621" />
Earlier I identified the format /assets/galleries/xxxx/_resampled/xxxx.jpg as worthy of investigation.
$ grep -o 'src="/assets[^"]*' *html
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-BAL-Busy2.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-Cheers.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-COV-BarBusy.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-FolkDrinking1.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-FolkDrinkingTWINS.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-HAM-Busy2.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-HAM-BusyGirls.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-HAM-GirlDrinking.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-PUT-HappyHour.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-ShakeShake.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-SOHO-BarBusy2.jpg
beatone.co.uk-.html:src="/assets/Banner-Images/Home/_resampled/croppedimage626168-SOHO-BarMixing.jpg
beatone.co.uk-.html:src="/assets/Widgets/_resampled/croppedimage158220-book-a-party.jpg
beatone.co.uk-.html:src="/assets/Widgets/_resampled/SetWidth182-book-a-party-title.gif
beatone.co.uk-.html:src="/assets/Widgets/_resampled/croppedimage158220-happy-hour.jpg
beatone.co.uk-.html:src="/assets/Widgets/_resampled/SetWidth182-happy-hour-title.gif
customcanvas.fritzandandre.com-.html:src="/assets/Banners/blueplanetvespa.jpg
customcanvas.fritzandandre.com-.html:src="/assets/Banners/scooter5.jpg
customcanvas.fritzandandre.com-.html:src="/assets/Banners/scooterad4.jpg
customcanvas.fritzandandre.com-.html:src="/assets/Banners/traffic.jpg
customcanvas.fritzandandre.com-.html:src="/assets/Banners/badboy.jpg
hungryhearts.no.html:src="/assets/Uploads/_resampled/croppedimage177207-hungry-heartsindex5.jpg
hungryhearts.no.html:src="/assets/Uploads/_resampled/croppedimage8075-kvad-cropst2web2.jpg
hungryhearts.no.html:src="/assets/Uploads/_resampled/croppedimage8075-IMG6686.jpg
hungryhearts.no.html:src="/assets/Uploads/_resampled/croppedimage8075-henri4.jpg
hungryhearts.no.html:src="/assets/Uploads/_resampled/croppedimage8075-bil4.jpg
...
$ grep -lo 'src="/assets.*_resampled' *html | wc -l
13
The pattern appears in only 13 of the samples. At first it doesn't appear to be a very unique match but a google query for "/assets/ _resampled/" returned almost entirely SilverStripe websites.
6. Review of unique patterns identified
=======================================
Pattern 1 - Meta generator tag
-------------------------------
Examples:
$ grep -hi 'name="generator' *html | sed 's/^[ \t]*//g' | sort -u
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />
The meta generator tag is most likely to be removed by a web developer. It sometimes has the version number which is useful.
I will give this pattern a certainty of 100%.
Pattern 2 - Cookie PastVisitor
-------------------------------
Googling for 'cookie "PastVisitor"' turns up results referring directly to SilverStripe and results referring to websites that turn out to be running SilverStripe. This cookie name, while generic sounding appears to be only used by SilverStripe and will make a good plugin match.
Examples:
$ grep -h PastV *meta
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/
I will give this pattern a certainty of 100% because I couldn't find any examples of a non-SilverStripe website using the cookie name.
Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css
-----------------------------------------------------------------
Many of the samples have at least 3 css files named layout.css, typography.css and form.css. The use of these names is not exclusive to SilverStripe and is considered best practice for making CSS frameworks but the order of their appearance combined with the folder structure is unique enough for a 'probable' plugin match.
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/layout.css?m=1266347738" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/typography.css?m=1266347623" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/form.css?m=1247030621" />
Not all samples have the ?m= after the css filename. Here are examples without the ?m=
$ grep -h -A 2 layout.css plugin-development/tests/silverstripe/*html |fgrep -v "?m="
<link rel="stylesheet" type="text/css" href="themes/dcsd/css/layout.css" />
<link rel="stylesheet" type="text/css" href="themes/dcsd/css/form.css" />
<link rel="stylesheet" type="text/css" href="tutorial/css/layout.css" >
<link rel="stylesheet" type="text/css" href="tutorial/css/typography.css" >
<link rel="stylesheet" type="text/css" href="tutorial/css/form.css" >
<link rel="stylesheet" href="/themes/firstgalaxies/css/layout.css" type="text/css">
<link rel="stylesheet" href="/themes/firstgalaxies/css/typography.css" type="text/css">
<link rel="stylesheet" href="/themes/firstgalaxies/css/form.css" type="text/css">
I will give this pattern a certainty of 75% because these three CSS filenames are considered best practice.
Pattern 4 - image assets url structure
-----------------------------------------------------------------
<img src="/assets/.*/_resampled/.*.jpg"
Examples:
<img src="/assets/magazine/sr6/toc/_resampled/croppedimage220165-02-Two Vietnam Vets.jpg" alt="" />
<img class="left noborder" src="/assets/Uploads/services/icons/fundraisers-icon.jpg" alt="Fundraisers Icon" />
At first it doesn't appear to be a very unique match but a google query for "/assets/ _resampled/" returned almost entirely SilverStripe websites.
I will give this pattern a certainty of 75% because I found at least 1 non-SilverStripe example with Google.
7. Write the plugin
=======================================
Build on the plugin template
---------------------------------------
The plugin template is found in the plugin-development/ folder. Copy this into the plugins/ folder with the name of your choosing. All plugin names have the .rb extension.
The template:
Plugin.define "Plugin-Template" do
author "Enter Your Name"
version "0.1"
description "Describe what the plugin identifies. Include the homepage of the software package"
examples %w| include-some.net example-websites.com here.com |
# a comment block here is a good place to make notes for yourself and others
# There are four types of matches: regexp, text, ghdb
# Matches are enclosed in {} brackets and separated by commas
matches [
{:name=>"a brief description of the match, eg. powered by in footer",
:probability=>100, # this isn't a real probability. 100 is certain, 75 is probably and 25 is maybe
:regexp=>/This page was generated by <a href="http:\/\/www.genericcms.com\/en\/products\/generic-cms\/">Generic CMS<\/a>/ },
{:name=>"title",
:probability=>75,
:text=>"<title>Generic Homepage</title>" }
]
end
Fill in the plugin name, author, version, description and examples fields. The examples are a ruby array delimited by whitespace and the http:// prefix is optional.
Plugin.define "SilverStripe" do
author "Andrew Horton"
version "0.1"
description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"
examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.clockwork.co.nz/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.naciondnb.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |
# a comment block here is a good place to make notes for yourself and others
# There are four types of matches: regexp, text, ghdb
# Matches are enclosed in {} brackets and separated by commas
matches [
{:name=>"a brief description of the match, eg. powered by in footer",
:probability=>100, # this isn't a real probability. 100 is certain, 75 is probably and 25 is maybe
:regexp=>/This page was generated by <a href="http:\/\/www.genericcms.com\/en\/products\/generic-cms\/">Generic CMS<\/a>/ },
{:name=>"title",
:probability=>75,
:text=>"<title>Generic Homepage</title>" }
]
end
Match Pattern 1 - Meta generator tag
----------------------------------------------------
Review the examples you have collected for match 1 and decide on what type of match is best suited to this pattern.
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />
Match types are:
regexp - Ruby regular expressions that start and end with / characters
text - A text string enclosed in ' or " characters
ghdb - Google Hacking Database. This uses some Google query parameters. It currently supports intitle:, filetype:, inurl:
For this plugin to match we will look for a meta tag with the name 'generator' and a content parameter that starts with "SilverStripe". Later we can extract the version number. A regular expression match is best suited for this.
The following regular expression will match the tag.
/<meta name="generator"[^>]*content="SilverStripe/
Notice how I haven't tried to match the http-equiv="generator" part of the tag or the website URL in the content field. Those parts of the tag are irrelevant and may change in future versions of SilverStripe.
Note: If you don't understand regular expressions you could make a text match like:
:text=>'<meta name="generator" http-equiv="generator" content="SilverStripe'
The plugin now looks like:
Plugin.define "SilverStripe" do
author "Andrew Horton"
version "0.1"
description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"
examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.clockwork.co.nz/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.naciondnb.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />
matches [
{:name=>"meta generator tag",
:probability=>100,
:regexp=>/<meta name="generator"[^>]*content="SilverStripe/}
]
end
I have included the meta generatator tag examples within the plugin as comments because this is a good place to refer to them later.
Plugin Testing
---------------------------------------
It's good practice to test your plugin while writing it to make sure it works.
If you want to ensure your plugin is loaded, run the following command:
$ ./whatweb -l
This shows all loaded plugins displayed along with the version number.
Test your current plugin on the SilverStripe samples you have collected. The whatweb parameters used are:
-v Verbose. This shows us which matches are being found
-p Plugins. Only load the SilverStripe plugin
$ ./whatweb -v -psilverstripe ./plugin-development/tests/silverstripe/*html
./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
Identifying: ./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html
HTTP-Status:
./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/beatone.co.uk-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/hungryhearts.no.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html []
Identifying: ./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html
HTTP-Status:
./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
We notice that charcoalinteriors.com.au and maungataniwha.co.nz aren't matching. After viewing the HTML files we notice they do not include the meta generator tag. Our first match is working correctly.
Match Pattern 2 - Cookie PastVisitor
------------------------------------------------------
Review the examples you have collected and decide on what type of match is best suited.
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/
Matching the cookie cannot be done within the matches array. Create a function called passive that will be triggered whenever the plugin is used.
def passive
m=[]
m
end
This code creates a empty array called m and returns the value of that array. This array will contain hashes in the same format as the matches section. Hash element names can be name, probability, version, etc.
def passive
m=[]
m << {:name=>"PastVisitor Cookie", :probability=>100 } if @meta["set-cookie"] =~ /PastVisitor=[0-9]+.*/
m
end
Now the function is checking the @meta array element "set-cookie" is see if it contains a regular expression that begins with PastVisitor= then has some numbers.
We cannot test this against the saved HTML files in our plugin-development/silverstripe/ folder. Instead we will test it using example sites. Note the whatweb parameter -e uses the examples in the loaded plugins as targets.
$ ./whatweb -v -psilverstripe -e
http://charcoalinteriors.com.au/ [200]
Identifying: http://charcoalinteriors.com.au/
HTTP-Status: 200
http://beatone.co.uk/ [200] SilverStripe
Identifying: http://beatone.co.uk/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
http://maungataniwha.co.nz/ [200]
Identifying: http://maungataniwha.co.nz/
HTTP-Status: 200
http://hungryhearts.no [200] SilverStripe
Identifying: http://hungryhearts.no
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
http://customcanvas.fritzandandre.com/ [200] SilverStripe
Identifying: http://customcanvas.fritzandandre.com/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
http://unbounded.org/ [200] SilverStripe
Identifying: http://unbounded.org/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100},
{:name=>"PastVisitor Cookie", :probability=>100}]]]
http://weonline.in [200] SilverStripe
Identifying: http://weonline.in
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
http://www.arprostatecancer.org/ [200]
Identifying: http://www.arprostatecancer.org/
HTTP-Status: 200
http://victoriaoruwari.com/ [200] SilverStripe
Identifying: http://victoriaoruwari.com/
HTTP-Status: 200
[["SilverStripe", [{:name=>"PastVisitor Cookie", :probability=>100}]]]
Our plugin has recognised the PastVistitor cookie for unbounded.org and victoriaoruwari.com so we know that it works.
Using just the two matches so far would be insufficient. Notice how some sites match only the meta generator tag, other match only the cookie and some aren't matched at all.
Match Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css
-----------------------------------------------------------------------
Review the examples you have collected and decide on what type of match is best suited.
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/layout.css?m=1266347738" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/typography.css?m=1266347623" />
<link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/form.css?m=1247030621" />
<link rel="stylesheet" href="/themes/firstgalaxies/css/layout.css" type="text/css">
<link rel="stylesheet" href="/themes/firstgalaxies/css/typography.css" type="text/css">
<link rel="stylesheet" href="/themes/firstgalaxies/css/form.css" type="text/css">
I decided earlier to give this pattern a certainty of 75% because using this set of filenames for CSS files is considered best practice. I also discovered an example where the typography.css file wasn't included. I chose to not match that because three names in order is a more likely unique match.
A regular expression is the best choice to match match these three css files in order:
/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/
Including this regular expression gives us the following matches array in our plugin. Notice how a comma is included after the first match.
matches [
{:name=>"meta generator tag",
:probability=>100,
:regexp=>/<meta name="generator"[^>]*content="SilverStripe/},
{:name=>"layout, typography, form css files",
:probability=>75,
:regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/}
]
Test the regular expression to make sure it works.
$ ./whatweb -v -psilverstripe ./plugin-development/tests/silverstripe/*html
./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
Identifying: ./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html
HTTP-Status:
./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/beatone.co.uk-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/hungryhearts.no.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100}]]]
./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html [] probably SilverStripe
Identifying: ./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>
/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/,
:name=>"layout, typography, form css files",
:probability=>75}]]]
./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:probability=>100},
{:regexp=>
/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/,
:name=>"layout, typography, form css files",
:probability=>75}]]]
3 of the first 5 didn't match the CSS files, charcoalinteriors.com.au, beatone.co.uk, and hungryhearts.no.html. I manually inspected the files to see what their CSS files were. None of them include the 3 CSS files in order so our regular expression works.
Match Pattern 4 - image assets url structure
-----------------------------------------------------------------
Review the examples you have collected and decide on what type of match is best suited.
<img src="/assets/magazine/sr6/toc/_resampled/croppedimage220165-02-Two Vietnam Vets.jpg" alt="" />
<img class="left noborder" src="/assets/Uploads/services/icons/fundraisers-icon.jpg" alt="Fundraisers Icon" />
This match is best found with a regular expression.
Earlier I thought it didn't appear to be a unique match but a google query for "/assets/ _resampled/" returned almost entirely SilverStripe websites. I'm giving this pattern a certainty of 75% because I found at least 1 non-SilverStripe example with Google.
The following regular expression will work, <img src="/assets/[^/]+/_resampled/[^"]+.jpg" . In plain english this is read as
<img src="/assets/anything but a slash/_resampled/anything but double quotes.jpg"
Our plugin matches array now looks like:
matches [
{:name=>"meta generator tag",
:probability=>100,
:regexp=>/<meta name="generator"[^>]*content="SilverStripe/},
{:name=>"layout, typography, form css files",
:probability=>75,
:regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/},
{:name=>"<img src="/assets/something/_resampled/something.jpg"",
:probability=>75,
:regexp=>/<img src="/assets/[^/]+/_resampled/[^"]+.jpg"/}
]
Next I tested the plugin to confirm that it matches some of the samples.
Test the plugin against all the examples
----------------------------------------
Test the plugin against all the example websites to test how effectively it idenfies them. Instead of testing against the saved HTML files I will test against the example URLs in the plugin so that the match that checks the cookies will work.
$ ./whatweb -psilverstripe -e
http://unbounded.org/ [200] SilverStripe
http://charcoalinteriors.com.au/ [200]
http://customcanvas.fritzandandre.com/ [200] SilverStripe
http://beatone.co.uk/ [200] SilverStripe
http://maungataniwha.co.nz/ [200] probably SilverStripe
http://hungryhearts.no [200] SilverStripe
http://www.benpearce.co.nz/ [200]
http://victoriaoruwari.com/ [200] SilverStripe
http://www.arprostatecancer.org/ [200]
http://weonline.in [200] SilverStripe
http://www.choidoco.com/demo/ [200] SilverStripe
http://www.cavendishimaging.com/ [200] SilverStripe
http://www.bradyinc.com/ [200] SilverStripe
http://www.clockwork.co.nz/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.chapmansurfboards.com/ [200] SilverStripe
http://www.fairtradenap.net/ [200] probably SilverStripe
http://www.executivemediasearch.com/ [404]
http://www.fuel.ie/silverstripe [301]
http://www.firstgalaxies.org/ [200] SilverStripe
http://www.fuel.ie/silverstripe/ [200] SilverStripe
http://www.enamaine.org/ [200] SilverStripe
http://www.frussian.com.ar/ [200] SilverStripe
http://www.infinitestillness.ie/ss [301]
http://www.holistichealth.com/ [200] SilverStripe
http://www.gsbc.edu/ [200] SilverStripe
http://www.hutmacherin.com/ [301]
http://www.infinitestillness.ie/ss/ [200] SilverStripe
http://www.hutmacherin.com/start [200] probably SilverStripe
http://www.latenightdisco.com/ [200] SilverStripe
http://www.kitesurfnelson.co.nz/ [200] probably SilverStripe
http://www.intandemtheatre.org/ [200] SilverStripe
http://www.moerakihavenmotel.co.nz/ [200] SilverStripe
http://www.lisamarieelliott.com/ [200] SilverStripe
http://www.moonlitekustoms.com/ [200] SilverStripe
http://www.naciondnb.com/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.maklerservice-greiz.de/ [200] SilverStripe
http://www.monjasantner.de/ [200] SilverStripe
http://www.nadabakery.co.nz/ [200] SilverStripe
http://www.rcaforum.org.nz/ [200] SilverStripe
http://www.moto-racepaint.com/ [200] SilverStripe
http://www.peterpanvakantieclub.nl/ [200] SilverStripe
http://www.robert80.de/ [200] SilverStripe
http://www.silverstripe.com/ [200] SilverStripe
http://www.stillrunnin.com/ [200] SilverStripe
http://www.textiprints.com/ [200] SilverStripe
http://www.thelightboxdesigns.com/ [200]
http://www.silverstripe.org.pl/ [200] SilverStripe
http://www.whileyouwait.co.nz/ [200] SilverStripe
http://www.tobychampion.co.uk/ [500]
http://www.wend.nl/ [200] SilverStripe
http://www.upstreamgroup.com/ [200] SilverStripe
http://www.verus.com.tr/ [200] SilverStripe
Most of the sites with a HTTP 301 status redirect to a page with a status of 200 and are identified as SilverStripe. Some sites are no longer active so I removed them from the examples list.
Of the 45 live websites, 43 are identified as SilverStripe. It is accurate to say our plugin, using only passive matches identifies about 95% of SilverStripe websites.
Extract version numbers
-----------------------
The meta generator tag sometimes contains version numbers which we want to detect.
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />
To extract the version number I need to write some custom ruby code in the passive function.
if @body ~= /<meta name="generator"[^>]*content="SilverStripe [0-9\.]+/
v=@body.scan(/<meta name="generator"[^>]*content="SilverStripe ([0-9\.]+)/)[0].to_s
m << {:name=>"meta generator version", :probability=>100, :version=>v }
end
This code checks that the regular expression that has the SilverStripe version within the meta generator tag is in the HTML body. If so, it copies it into a variable called v then includes it in a hash which is put into the array of matches.
Testing the code shows that version numbers are being extracted. Note that if SilverStripe were to include letters after version number, eg. 2.3.5b that the letter wouldn't be recognised.
$ ./whatweb -psilverstripe ./plugin-development/tests/silverstripe/*html
./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html [] probably SilverStripe
./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/victoriaoruwari.com-.html []
./plugin-development/tests/silverstripe/weonline.in.html [] SilverStripe[2.3.1]
./plugin-development/tests/silverstripe/unbounded.org-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.benpearce.co.nz-.html []
./plugin-development/tests/silverstripe/www.choidoco.com-demo-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.bradyinc.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.cavendishimaging.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.chapmansurfboards.com-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.arprostatecancer.org-.html []
./plugin-development/tests/silverstripe/www.executivemediasearch.com-.html []
./plugin-development/tests/silverstripe/www.enamaine.org-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.firstgalaxies.org-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.fairtradenap.net-.html [] probably SilverStripe
./plugin-development/tests/silverstripe/www.fuel.ie-silverstripe.html [] SilverStripe
./plugin-development/tests/silverstripe/www.holistichealth.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.hutmacherin.com-.html [] probably SilverStripe
./plugin-development/tests/silverstripe/www.gsbc.edu-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.clockwork.co.nz-.html []
./plugin-development/tests/silverstripe/www.frussian.com.ar-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.infinitestillness.ie-ss.html [] SilverStripe
./plugin-development/tests/silverstripe/www.lisamarieelliott.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.latenightdisco.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.kitesurfnelson.co.nz-.html [] probably SilverStripe
./plugin-development/tests/silverstripe/www.intandemtheatre.org-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.moerakihavenmotel.co.nz-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.maklerservice-greiz.de-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.monjasantner.de-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.naciondnb.com-.html []
./plugin-development/tests/silverstripe/www.moto-racepaint.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.nadabakery.co.nz-.html [] SilverStripe[2.3.1]
./plugin-development/tests/silverstripe/www.moonlitekustoms.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.peterpanvakantieclub.nl-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.robert80.de-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.rcaforum.org.nz-.html [] SilverStripe[2.3.0]
./plugin-development/tests/silverstripe/www.silverstripe.org.pl-.html []
./plugin-development/tests/silverstripe/www.thelightboxdesigns.com-.html []
./plugin-development/tests/silverstripe/www.silverstripe.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.textiprints.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.stillrunnin.com-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.tobychampion.co.uk-.html [] SilverStripe
./plugin-development/tests/silverstripe/www.upstreamgroup.com-.html [] SilverStripe[2.3.1]
./plugin-development/tests/silverstripe/www.whileyouwait.co.nz-.html [] SilverStripe[2.0]
./plugin-development/tests/silverstripe/www.wend.nl-.html [] probably SilverStripe
./plugin-development/tests/silverstripe/www.verus.com.tr-.html [] SilverStripe
We can see that only some websites include the version number in the meta generator tag.
The final plugin
----------------
Plugin.define "SilverStripe" do
author "Andrew Horton"
version "0.1"
description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"
examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />
matches [
{:name=>"meta generator tag",
:probability=>100,
:regexp=>/<meta name="generator"[^>]*content="SilverStripe/}, #" I have included a comment with double quotes for the benefit of syntax hilighting in gedit
{:name=>"layout, typography, form css files",
:probability=>75,
:regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/},
{:name=>'<img src="/assets/something/_resampled/something.jpg"',
:probability=>75,
:regexp=>/<img src="\/assets\/[^\/]+\/_resampled\/[^"]+.jpg"/} #"
]
# Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
def passive
m=[]
m << {:name=>"PastVisitor Cookie", :probability=>100 } if @meta["set-cookie"] =~ /PastVisitor=[0-9]+.*/
if @body =~ /<meta name="generator"[^>]*content="SilverStripe [0-9\.]+/
v=@body.scan(/<meta name="generator"[^>]*content="SilverStripe ([0-9\.]+)/)[0].to_s
m << {:name=>"meta generator version", :probability=>100, :version=>v }
end
m
end
end
8. Closing Notes
=======================================
I have shown you the process of how to write a simple WhatWeb plugin. The most important part of the process is the complete research that happens before writing any matches.
Some people will be tempted to write a pattern for the meta-generator tag then stop. Such an approach would identify about 75% of SilverStripe sites. Futhermore there is a generic meta generator tag plugin so such an effort would be of little practical use.
Our final plugin identifies about 95% of SilverStripe websites using only passive matches. An aggressive plugin that guesses URLs would increase the effectivenses to 100% however aggressive plugins are not stealthy, they use more bandwidth and so are less suitable for large scale website identification. However aggressive plugins are useful during penetration testing to identify frameworks. Writing aggressive plugins is a more advanced topic and will be covered in another tutorial.
Please submit your plugins to andrew [at] morningstarsecurity.com to be included in the next release of WhatWeb.
9. Resources
=======================================
The best way to learn how to develop plugins is by reading the plugins bundled with WhatWeb.
To learn and test regular expressions visit: http://rubular.com/
WhatWeb homepage: http://www.morningstarsecurity.com/research/whatweb
Visit MorningStar Security for the best Information Security news at http://www.morningstarsecurity.com/news
