Spurious correlation ...

Go here ... http://www.tylervigen.com/

[Thank you, John Gandar, for the link.]

I've found it ! (again)

Data scientists like me spend a lot of time worrying about which type of model to fit and the best way of fitting it. 

As one example, we think about regression, its many pitfalls, and its arguable overuse in market research, and we look for alternatives that deal with issues such as collinearity, non-linearity in relationship, non-numeric dependent variables, non-‘normal’ predictors, and missing values, not to mention data files that are invariably imperfect (e.g. when a value of zero can mean either ‘0’ or ‘Don’t know’ or ‘the question wasn’t asked of that respondent’).

In fact, the frontier for modelling in market research is now very much associated with approaches from the data mining field, such as (you know I am going to say this) Random Forests and even Conditional Random Forests. 

Random Forests and similar approaches (which together fall into the category of ‘ensemble modelling’, which is more or less the quantitative version of Wisdom of Crowds) form the backbone of competitive attempts to predict the near-impossible, via the various competitions on Kaggle http://www.kaggle.com/.

In my search for alternatives, a few years back I came across the unfortunately spelt ‘Eureqa’ software, and have written about it previouly. 

Eureqa Desktop uses something called ‘Symbolic Regression’, and its mode of operation is essentially to conduct a search for the best model amongst all possible models.  That is, it doesn’t just say “Here is the model you’ve asked for and I will now calibrate it”.  It says “Here are a zillion possible different models, and I will calibrate them all and let you know which is the best one.” 

Eureqa Desktop is available from http://www.nutonian.com/ and is, unfortunately, no longer free. 

However, the Excel version is (at the time of writing) in beta testing, and is free.  And unlike Eureqa Desktop, Eureqa for Excel pretty much makes all the decisions for you (such as what functional forms to allow in the models that it examines).

I’m still to be convinced about Eureqa … but I do understand, for example, that if you input the co-ordinates of a swinging pendulum, Eureqa will eventually come up with Newton’s gravitational law as the best model for the data !

However in market research, we rarely have data sets that display the same degree of precision as a swinging pendulum, and finding a model that predicts satisfactorily, however good the software used, is probably always going to be at least a little problematic.

Ultimately it comes down to finding an approach that (a) may not be perfect but works well enough and (b) the outputs of which make sense, in the light of what else we know about the particular situation being analysed.

As the statistician G.E.P. Box wrote in 1979 … “All models are wrong, but some are useful.” 

Something in the air ?

Like very many people in this industry, I have loads of stuff that I can’t afford to lose.

So the only option is to back it up, both frequently and in full.

What’s the best way to do that?  Well, you’ll all have your own strategies.

For example, one of my clients was telling me that she backs up all her computer-based records on to a hard drive at the end of every day, then one of the people in her office takes the drive home and looks after it overnight.  That way, by alternating between two or more hard drives, she is pretty sure of never being more than a day or two behind, if disaster strikes.

I do something similar, but instead using an NAS (network attached storage) hard drive (i.e. a 4Tb drive that is accessed through my office network) and backing up to that automatically every night at 1am.  The NAS drive is in a “secret” place, that no-one (I hope) will ever find.

But that’s not good enough.  So I’ve also been a user of cloud storage for a couple of years now, starting with Google’s service.  But ultimately, I opted for Microsoft’s ‘Onedrive’ service (the product formally known as ‘Skydrive’, but which was apparently the subject of an infringement action in the UK by Sky TV, so the name had to change).   I pay $USD100 per year for 200Gb of cloud storage capacity, which is a pretty fair charge, particularly when compared with Google’s fees for a similar, but lower capacity, cloud storage service.

Now, you’d think that Microsoft would have this pretty well-sorted. 

Unfortunately not.

For example:

• Some Onedrive users (particularly Windows 8.1 users) have had massive problems in just trying to access a local copy of a file they have put onto the cloud service via syncing from another PC.
• It seems also that a move to Windows 8.1 can mean that the service completely stops.  One of the solutions to this suggested by the Microsoft help desk was to back up your entire Onedrive directory, then delete it, then re-make it and start again, whilst at the same time saying the Microsoft took ‘no responsibility’ for any loss of data.  [Really?  I thought the whole purpose of cloud storage was so that you wouldn’t lose data.]
• In my case, I have around 130Gb of data in my Onedrive storage, and re-syncing that to the cloud facility literally takes days.
• Whilst you can upload files with path lengths greater than 255 characters, once you do it, when syncing, the service simply stops at the point where it encounters the ‘excessive’ path length, and the only fix is to log onto the service on-line, locate the offending file and path, shorten it, then re-start the syncing.  But if there is another path with the same problem, the procedure starts all over again.  Some users have reported having literally hundreds of long-path-name files which they have had to rectify one by one.
• When you are not on a wireless connection, but instead on 3/4G, it seems that Onedrive becomes extremely reluctant to be helpful, on the grounds that it might use too much of your bandwidth.

·         Lastly, if you want further indication of difficulties encountered by Onedrive users, just search “skydrive-onedrive-error-files-cant-be-uploaded”, or similar.

Having said all that (sorry for the rant), when Onedrive is working, it works really very well, and pretty much seamlessly.

But also bear in mind that Microsoft has some pretty severe conditions for using their service … http://wmpoweruser.com/watch-what-you-store-on-skydriveyou-may-lose-your-microsoft-life/

I haven’t anywhere near the same experience with other cloud storage facilities, so can’t comment on those.

Regardless, and in summary, I think the moral is never to rely on just one form of back-up.  If I had to re-create even just this article, it would mean at least another hour out of my day.  Imagine if I had to re-create all of yesterday’s work?

How not to respond to a customer suggestion ...

Did Virgin actually read my suggestion? ... (hint: read from the bottom)

Membership number: 2101*****2

Dear Mr Maclean

Thank you for taking the time to contact us about your feedback.

I understand you may be disappointed and on behalf of Velocity Frequent Flyer, please accept my sincerest apologies.

Receiving feedback from our members ultimately helps us to improve our program and we appreciate you taking the time to provide us with your feedback.

For further information about Velocity Frequent Flyer or your own account, please visit our website: www.velocityfrequentflyer.com or call the Membership Contact Centre on the number below.

Yours sincerely

Shanne Mcman
Velocity Frequent Flyer
____________________________________
Velocity Membership Contact Centre:
Australia - 131 875
New Zealand - 0800 230 875
Everywhere else - +61 2 8667 5924
(Monday to Friday from 8am to 8pm - Sydney time)
___________________________________________
This message is being sent to you in response to an enquiry you addressed to Velocity Frequent Flyer, and is being sent by Velocity Rewards Pty Ltd ABN 98 116 089 448 of PO BOX 1034 Spring Hill 4004.

---------------------------------------------
Original Message:
---------------------------------------------
Subject : RED - 2101*****2 - Suggestion - Other
From ID : 8650013783
Name Surname : QMAN SYDNEY
From Email : info@velocityfrequentflyer.comDate : 2014-04-15 13:49:54.0
---------------------------------------------
QMAN SYDNEY wrote:
Submitted on: 15 Apr 2014 - 13:49
First Name: Alan
Last Name: MacLean
Email Address: scott@nulinkanalytics.com.au Phone: 0419504588
Velocity Member: Yes
Velocity Number: 2101*****2
Velocity Tier: R
Type of Feedback: Suggestion
Relating To: Other
Reference Number: Not Entered


Summary: Lounge access


Message: Hi. I am a Lounge member, and have been for some years, at an annual cost of $400 . 

My suggestion is that you give Lounge members access to the priority check-in queue at the general check-in area. 

I'm pretty sure I can use the AirNZ priority check-in queue when flying with them, by virtue of my Virgin Lounge membership, so it seems inconsistent that I cannot have the same when flying Virgin.

omg ... they are all losing the plot

Why do these utility/services/bank/whatever companies make it sooooooo hard ?

Why do I have to phone them ?  Why can't they phone me ?

Smart meters from a not so smart company ?

Even their own feedback page has an error ...

And just have a look at the response to my complaint .... I've pasted this in at the bottom.

In particular, why can't they call me ?  I had to give them my phone number when I lodged my feedback.  Who's the customer here ?

Update: 27nov2013 - I gave in and called 'the number provided on the letter' and was told that the serial number is the same as the NMI number on my usual electricity bill, but with one extra (apparently secret) digit at the end.  Then I attempted to get on the relevant website to register for 'Energy Easy' and received this response:

This webpage is not available

The webpage at https://energyeasy.ue.com.au/ might be temporarily down or it may have moved permanently to a new web address.

Here are some suggestions:

• Reload this web page later.

Error 118 (net::ERR_CONNECTION_TIMED_OUT): The operation timed out.

At this point, I gave up.

-------- Original Message --------

Subject:	United Energy Service Desk Email Address Change
Date:	Wed, 20 Nov 2013 07:02:30 +0000
From:	UCMS AAM Service Desk <MailboxAccountUCMSAlintaServiceDesk@jemena.com.au>
To:	scott@nulinkanalytics.com.au <scott@nulinkanalytics.com.au>

Please be advised that the United Energy Service Desk email address has changed to:  ueservicedesk@ue.com.au

Please ensure that your records are updated to the new United Energy contacts as per the ‘Retail Operations Contact List’ (ROCL).

Your email has been forwarded to the new email address on your behalf.


-------- Original Message --------

Subject:	RE: Feedback
Date:	Thu, 21 Nov 2013 03:19:34 +0000
From:	ueservicedesk <ueservicedesk@ue.com.au>
To:	'scott@nulinkanalytics.com.au' <scott@nulinkanalytics.com.au>

Hi Scott,

Please be advised to call the number provided on the letter.


Cheers,
Rein

Reneir Racaza
Service Desk Consultant
United Energy Service Desk

United Energy and Multinet Gas
T:  (03) 92565249
E: ueservicedesk@ue.com.au

Unable to lead a shelta-ed existence ...

Update (May 2014)
I bought another Shelta umbrella ... big mistake, since that one also broke after only a couple of uses.  And they still want me to make a two hour round trip to get it replaced.  Never again - from now on I'm buying cheap Chinese umbrellas (which is probably where Shelta gets theirs made anyway) and if they break, at least I'll only be out of pocket by few dollars instead of over $30.

Original post
See below ... love the customer service (not) ...

But there is a happy end to this story.  

I decided that a round trip of 2 hours approx to take the faulty item back to the manufacturer/importer was not something I wanted to do, more so given that I don't actually have the receipt.

So I hunted in ebay for a replacement and found one.  But I messaged the seller and asked her to open up the umbrella first, to make sure it was OK, since my experience with Shelta had so far been not exactly positive.

And to my extreme surprise, the seller replied, firstly apologising for taking a whole day to get back to me (it was the weekend for goodness sake - I certainly didn't expect any sort of response for a few days at least) and secondly, to say that if I got the faulty item back to her with $7 to cover postage, she would replace it free of charge.  Now remember, this is a seller from whom I not actually bought anything, ever.

How about that for knock-out customer service ?

Here is her ID for anyone interested ... balloonsandstuff



-------- Original Message --------

Subject:	RE: Faulty umbrella
Date:	Wed, 13 Nov 2013 10:48:34 +1100
From:	Diane Stephens <d.stephens@shelta.com.au>
Reply-To:	<d.stephens@shelta.com.au>
Organization:	Shelta
To:	<scott@nulinkanalytics.com.au>

Good Morning

Sorry to hear about your umbrella, as you are unsure of the place of purchase, we would require that you bring or send you umbrella back to Shelta in Hallam

For assessment/repair.  Please note that the terms of our warranty require you to produce your proof of purchase/receipt for warranty claims

Regards

Diane Stephens

2/25 Conquest Way

Hallam Vic 3803

Phone: 03 9702 3377

Fax:       03 9708 6779

From: Scott MacLean [mailto:scott@nulinkanalytics.com.au]
Sent: Tuesday, 12 November 2013 4:34 PM
To: p&hvic@shelta.com.au
Subject: Faulty umbrella

Hello.

I received a nice new Shelta Umbrella for my birthday this week.

But the very first time I used it (today), one of the struts was bent and it wouldn't open properly.

It wasn't windy, and I got quite wet walking home from the station.

I have no idea where it was purchased, except that it was somewhere in Melbourne.

May I have a replacement please?

My address is:

Unit 3,
49 Head Street,
Brighton VIC 3186.

The design is the cats/piano keys black and white theme.

Kind regards,

Scott MacLean

 

    Scott MacLean, Director

      ABN 24 772 951 133

    T: +61 419 504 588

    E: scott@nulinkanalytics.com.au 

   W: www.nulinkanalytics.com.au

    B: http://nulinkanalytics.blogspot.com.au/