Author Robert Hoff
Email hoff.rj@gmail.com
Updated 28 November 2018

  Projects   3D World   OpenGL   Physics   Charting   Web-work   Graphics   Uni-reports   Personal

 

Data collecting (web-parsing) and charting

Charts show datasets collected from the website Crowdcube.com

The charts are investement data derived from different company profiles (startups) during their campaigns on the crowdfunding platform Crowdcube.com. The timelines on all charts are drawn to match the period 5 Jan 2016 to 3 Feb 2016.

The Ethos profile (a health studio - website, a brief on the project is also available - investment brief) has the most interesting activity as being the main focus of the project. The other profiles have missing intervals being added in hindsight and dependent on available data (due to their campaign start and end times).

The data was sourced by polling the html, every few minutes, from respective profiles. Here showing a representative display of such data as it appears in the browser.

 

 

The central figure is the total money raised, in this example £558,400. Together with other data points; "investors so far" and "last investment" a transcript of events can be made.

Example

    06-Jan-2016 04:08:47  t=1.17  raised(£)=  63,610  amount(£)=   1,000  inv_c=27
    06-Jan-2016 09:26:52  t=1.39  raised(£)= 163,610  amount(£)= 100,000  inv_c=28
    06-Jan-2016 14:28:03  t=1.60  raised(£)= 163,620  amount(£)=      10  inv_c=29
    06-Jan-2016 14:41:59  t=1.61  raised(£)= 173,620  amount(£)=  10,000  inv_c=30
    06-Jan-2016 15:28:00  t=1.64  raised(£)= 188,620  amount(£)=  15,000  inv_c=31
    06-Jan-2016 15:40:58  t=1.65  raised(£)= 188,720  amount(£)=     100  inv_c=32
    06-Jan-2016 16:24:04  t=1.68  raised(£)= 188,770  amount(£)=      50  inv_c=33
    06-Jan-2016 17:43:30  t=1.74  raised(£)= 189,770  amount(£)=   1,000  inv_c=34
    06-Jan-2016 18:16:35  t=1.76  raised(£)= 190,770  amount(£)=   1,000  inv_c=35
    06-Jan-2016 19:12:49  t=1.80  raised(£)= 191,020  amount(£)=     250  inv_c=36
    06-Jan-2016 20:44:08  t=1.86  raised(£)= 191,030  amount(£)=      10  inv_c=37

Derived charts

Charts produced from the Ethos data; the line chart with investor volumes, a pie-charts of investment amounts and a breakdown of investment totals across each day.

Sourcefiles

Archive chart-demo.zip
Implemented in Java with JSoup (html parsing) and JFreeChart (charting library)

src > main >
  java > charting >
                App.java          <-- application entry, either for collecting data, parsing data or generating charts
                Chartview.java    <-- Jframe for displaying charts
                T.java             <-- 'T' for token, an enum holding constants of different profiles
                Timer.java        <-- A utility class for parsing and displaying dates
                analyse >                <-- the following classes each generating a different type of chart
                        BarChartDays.java
                        BarCharts.java
                        CombinedCharts.java
                        LineChart.java
                        PiechartDays.java
                        PiechartInvAmounts.java
                data >
                        EventData.java           <-- interface to the data collected, together with ParseFile and EventItem
                        EventItem.java
                        ParseFile.java
                        WebCollect.java          <-- the Jsoup web scraper, returns the result as console output
                                                            which is appended to the data files below using the
                                                            by running a cron-job on the batch script get-data.sh
data >                      <-- collected data saved as text files here as "inv-data_*.txt"
        inv-data_ethos.txt
        ...
get-data.sh   <-- batch script allowing a cron-job to run the scraper at repeated intervals

 

© Robert Hoff