Experiment Logs

This page logs the runs of each experiment set run, with any special notes about the run. The aim of this is so that I can look back at what I have been doing, and quickly see the path that I have taken.

Date of experiment start, what the experiment was, why it was being run, the outcome of the experiment and whether there were any problems. This log was started on 19 October 2011, and so any logs before this are made from memory.


 

important error in config detected: 25 July 2012:  I realised today that the mit-nov dataset had an incorrect end date in the all-datasets.xml config file (it was running until the end of the dataset!). This affects a large amount of stuff including edge-list generation including thresholding and bubbleH bubble runs, benchmarks etc. etc. This is not good. But I have updated the config file, and will re-run for the final evaluation.

FINAL_EVALUATION –

EXHAUSTIVE (part 3) – 17 Jan 2012 – Multiple Datasets: cambridge,enron,enron-a,enron-b,hypertext2009,infocom-2005,infocom-2006,mit-all,mit-nov,mit-oct,social-sensing and Multiple CFAs: KCLIQUE,HGCE,InfoMap,InfoHierMap,LinkClustering,Moses,Blondel,Conga,Random. Notably, Moses did not return any communities for either InfoCom dataset, and the MIT-ALL dataset, I will have to discuss this with Aaron at some point, it’s probably because all nodes are connected, and therefore there is no distinction between them. The run finished well, only Studivz and Salathe-School remain to be run, but they are proving troublesome….  results of this run are here

EXHAUSTIVE (part 2) – 13 Jan 2012 – Blondel, Conga and InfoHierMap are now implemented, and so this is an exhaustive run across all datasets, and all CFAs – Social Sensing, MIT-Nov, InfoCom-2005, InfoCom-2006, Hypertext2009, Enron, Cambridge. It seems a bug has been introduced at some point, this is suggested by the results for InfoMap, which be the same for Bubble and BubbleH, but for some reason, they are different. I have a suspicion that perhaps Communities are not being picked up correctly… UPDATE: it seems this phenomenon only occurred in mit-nov, where there was an issue with the community creating method. NaN’s were being reported instead of Hui betweenness in the community files. However, this may not have fixed the issue, so I am re-running all mit-nov datasets.

EXHAUSTIVE (part 1) – 4th Jan 2012, Exhaustive run of all CFAs and Datasets: cambridge,hypertext2009,infocom-2005,infocom-2006,mit-nov,social-sensing,salathe-school with CFAs: KCLIQUE,HGCE,LinkClustering,InfoMap,Moses,Blondel,Random. The idea is to get a current view of everything, and to see how Blondel (and CONGA when implemented) will match up. Future runs should be made when Random (multi-run with avg) and Conga have been implemented. Also, need to run for multi sub0versions of STUDIVZ dataset. Outcomes: InfoCom-2005 caused problems, as Moses did not produce any communities, which messed up the simulations.

BLONDEL_TEST – testing the Blondel implementation

DATASETS_STATS – testing the Salathe-School dataset

WWIC2012 Re run of all datsets and algorithms for the WWIC2012 paper, to ensure all data is correct.

November 30th 2011 – SELECTED_NODES – using only the pairs that have communicated during the training period, to determine the pairs of messages transmitted in the test period. Also, ensuring that the same HGCE output is used for both BubbleRAP and BubbleH. Also making sure results plots are generated based on a chosen metric (i.e. the best delivery ratio is used to determine plots for all other metrics) results here

SELECTED PAIRS

For enron, we will pick the paird of nodes that have communicated during the training periods, and send messages between only them during the test period. We still train the CFAs using all nodes. (to enable the ability for intermediate nodes to deliver messages to shine through). Initial results look good – now I need to create the relevant code for all routing algorithms (I had some code already for the BubbleH implementation)

ENRON_SPLIT

17 Nov 2011 – running a split version of Enron dataset – from 1st April to 1st July, with training periods in the first month. Values for ENRON training period:

  • Precision: 7
  • MEAN Average: 3.5013057e-6
  • MEDIAN Average: 1.6049000e-6
  • Pareto 80/20 : 4.0123000e-6
  • Reverse Pareto 20/80 : 8.0250000e-7
  • K = 3

CURRENT_SPLIT

11/11/11 Again, trying to get a clean run of both hypertext2009-split, and studivz-split. The idea being that I will then move onto running multiple random node allocations for studivz. e.g. 10 x 10 2 0 0 in studivz. Sucess, all runs completed, with only KCLIQUE needing to be re-run. (need to check the community hierarchy for HGCE in Studivz, as there might not be any!). Threshold Values: MEAN, MEDIAN, Pareto, R-Pareto 0.0035760146,0.0007309433,0.0035328924,0.0003654716. K=4

Studivz-three-split

the dataset is bigger than I thought- I had previously mis-represented the dates, (i think the unlimitedflood run on the whole dataset failed at some point…. anyway, the point it, the previous post about the activity in the dataset is wrong, there is a peak of 35000 posts per week in december 2007. So I will pick the period 1st October 2007 ro 1st Feb 2008 as the period, and pick nodes appropriately, using a month for training, and the rest for testing: 1st October to 31st October training;  1st November to 1st Feb as testing.

hypertext-split

Hypertext2009 split into three parts – day 1 is for training and days 2 and 3 are for testing. Outcome to be reported.

3rd November CURRENT

For the Hypertext2009 dataset, I set about running all CFAs over it, (including random) and seeing how they perform, parameters are default, and where applicable K=4 and Threshold is the 80th Percentile (0.0010406856). The aim is to see how this performs. We are hoping to find something interesting with regards to overlap and hierarchy.

2nd November Hypertext2009

Pádraig suggested this dataset, and so I spent some time adding it to the simulator, however on initial tests, the same error has occurred in the Studivz dataset…. and perhaps with fresh eyes I was able to find the problem. Embarasingly it WAS a config error afterall, the reference to the  communitydatasetloader was missing (in the All-communities.xml file)… so back to the plot!

25 Oct 2011

Trying to find the bug, exploring the possibility that Bubble is using the wrong folder for communities, or that for some reason there are no communities being used at all…. checking config, and src files.

24 Oct 2011 TESTSF5200_2

Re running TESTSF5200 just to make sure…. results show the same as before. BubbleRAP gives exactly the same results.

24 Oct 2011 TESTENRON

Testing a complete run of the Enron dataset as above, to see if the phenomenon relates only to the Studivz dataset. It turns out that this phenomenon does not appear. So there must be a problem with the Studivz dataset

21 Oct 2011 TESTSF5200

Re-run of STUDIVZ_THREE to make sure it wasn’t a strange bug for that run only, unfortunately the results are the same. I cannot find the reason for it just yet. Identified a bug in the moses config – a task was not running (to create the BubbleH JSON community structure). Also, server crashed some time after the run completed, but data was still there.

20 Oct 2011 STUDIVZ_THREE

Dataset: Studivz three month dataset, (2006-10-01 to 2007-02-01) using 5 random seeds, with 2 levels of depth – studivz-3month-5_2 – this experiment is to see how well each performs on a smaller section of the dataset – there is still a large number of nodes, but the time period is shorter. In this particular set there is some partition on the graph. This strangely has resulted in all BubbleRAP results being exactly the same! This is very strange,  but could be down to a config change, as the community finding seems to have created different sized communities.

16 Oct 2011 – STUDIVZ_EXPLORE

Dataset: Studivz 4 2 0 0, Experiment Group: STUDIVZ_EXPLORE, multiple runs for KCLIQUE with various parameters for MCT, parameters were chosen as follows: Take the edge list, and collect all of the connected times, take the MEAN average, the MEDIAN average and the 80th Percentile (as in Pareto’s 80/20 principle).  Parameters were initially: 0.0, MEAN: 0.0000041718,MEDIAN: 0.000015466502596778, 80pc: 0.0000003152, but after the firt run, I realised the 80pc value was wrong, so I added 80pc 0.0000164647.

16 Oct 2011- MOSES_ TEST

Dataset: Studivz 4 2 0 0, Experiment Group: MOSES_ TEST, run for Moses CFA, the idea was to see what results Moses comes up with on this dataset for BubbleH and BubbleRap. The run took around a day to process, the results were copied to expgroup: STUDIVZ422, so that plots could easily be made in comparison.

13 Oct 2011 – STUDIVZ422

Dataset: Studivz 4 2 0 0, Experiment Group: STUDIVZ422, a clean re-run of STUDIVZ_4_2_0_0 to make sure the results were the same. KCLIQUE,HGCE,InfoMap,LinkClustering for BubbleH and BubbleRAP, along with Unlimitedflood,

Oct 2011 – STUDIVZ_3_2_0_0 and STUDIVZ_4_2_0_0

Dataset: Studivz 4 2 0 0, Experiment Groups: STUDIVZ_3_2_0_0 and STUDIVZ_4_2_0_0 – using a sub set of the Studivz datasets, which uses 3 or 4 seed nodes, and finds build the network 2 hops out on the connected time graph, this experiment was designed to see how BubbleH and BubbleRap perform on this new dataset. KCLIQUE,HGCE,InfoMap,LinkClustering for BubbleH and BubbleRAP, along with Unlimitedflood, Hold and Wait, and SimpleVectorClock (4,2,0,0 only).

Sep 2011 – DATASET_EXPLORE2

re-running community finding, BubbleH and BubbleRAP on MIT-NOV, enron and Cambridge datasets with multiple paramters. The parameters given to CFAs were based on the best results of the the KCLIQUE algorithm. This time with automatic best run finding and plotting.  Best runs for each metric are stored: i.e. Delivery Ratio (max), Cost (min), Latency (min), Average Delivered Hops (min) and Average Undelivered Hops (min). The picking algorithm takes all results from a run with multiple parameters, and takes the last value recorded for the run and that metric, and stores the one with the best value. This means a run with multiple parameters can report each metric seperately, based on the best of each paramter run (e.g. best Cost my not be found in the same run as the best Delivery Ratio) – be careful of this in future!

August 2011 – DATASET_EXPLORE

re-running community finding, BubbleH and BubbleRAP on MIT-NOV, enron and Cambridge datasets with multiple paramters. The parameters given to CFAs were based on the best results of the the KCLIQUE algorithm. Someway through there were problems with some of the runs, so this experiment could not be relied upon

July 2011 – ENRON_EXPLORE

Exploring a range of parameters for the Enron dataset, for HGCE, InfoMap, KCLIQUE and LinkClustering – Some information here: http://mattstabeler.co.uk/phdblog/archives/1627

July 2011 – SSEXPLORE

Exploring a range of parameters for the Social Sensing dataset, for HGCE, InfoMap, KCLIQUE and LinkClustering

May 2011 – NEMO

Using Cambridge, InfoCom2005, InfoCom2006 and MIT-NOV, this experiment followed on from the ICWSM paper, using more datasets – however, we discovered that there was a problem with the results (based on a flaw in logic), and we had to abandon the experiment.




  1. No comments yet.
  1. No trackbacks yet.