@1stone opened this Issue on August 6th 2015

Since one of the last upgrades (I guess it started with the first 2.14 release) I noticed that the log-importer has failed to import any provided Apache logs.
After some tedious debugging, I believe I've tracked it down to a problem in the tracking api, which is not accepting the presented token_auth and thus does not honor the cip and cdt attributes in the request.
As a consequence of that, logs are handled with visitor_ip = local_ip, whereas the latter is contained in the exclusion list.

But back to the actual problem:
With tracking debug enabled, a request of

{"requests": [
    {"apiv": "1", 
     "url": "http://www.example.com/some/where", 
     "cdt": "2015-07-14 02:27:53",
     "cip": "11.249.64.238", 
     "rec": "1", 
     "idsite": "18"}, 
    {"apiv": "1", 
     "url": "http://www.example.com/some/where", 
     "cdt": "2015-07-14 02:27:56", 
     "cip": "11.249.64.238", 
     "idsite": "18"}
],
 "token_auth": "1ea06bef5839065d1159d8f5a941d40a"}

results in this response

Debug enabled - Input parameters: 
array (
)
Custom timestamp is 2055642 seconds old, requires &token_auth...
WARN: Tracker API 'cdt' was used with invalid token_auth
Current datetime: 2015-08-06 21:28:35
WARN: Tracker API 'cip' was used with invalid token_auth
DoNotTrack header not found
Visitor IP (was: 10.8.0.22) has been anonymized: 10.8.0.0
Visit level Custom Variables: 
...

The presented token_auth value is definitely a valid token from an user with superuser_access=1.
Testing it with other qualified tokens resulted in the same problem.

This is with Piwik 2.14.3. Please advise.

@tsteur commented on August 7th 2015 Owner

This reminds me of #7202 that we should not fallback to wrong data but instead not track at all in case the token_auth was not considered valid, this way such errors are detected earlier.

Did you modify your token? If not, please change your password so a new token will be generated.

I think we should first figure out whether really the bulk API is the problem or the importer. How did you execute the bulk request? I tried the following and it worked:

curl -i -X POST -d '{"requests":["?idsite=1&url=http://example.org&action_name=Test bulk log Pageview&rec=1&cip=10.10.10.1","?idsite=1&url=http://example.net/test.htm&action_name=Another bulk page view&rec=1"], "token_auth": "122bf5fffed14444444d93750054d3ed"}' http://apache.piwik/piwik.php

Are you using any custom plugins?

@1stone commented on August 7th 2015

The token was not modified, and it doesn't work with any superuser tokens.
I changed the password and retried with the new token, but no success either.

Apart from the default plugins, only the vanilla LdapLogin (v1.3.4) plugin was installed.

When trying to reproduce the error, please assure to pass the cdt and cip attributes, since they require the token_auth!

Please use something like this to reproduce the error:

curl -k -i -X POST -d '{"requests": [
  {"apiv": "1", 
    "url": "http://www.example.com/some/where", 
    "cdt": "2015-07-14 02:27:53", 
    "cip": "66.249.64.238",  
    "rec": "1", 
    "idsite": "6"}
], "token_auth": "1ea06bef5839065d1159d8f5a941d40a"}' https://apache.piwik/piwik.php

(Note: This is an extract of the request which I captured from the log-importer.py when it was send to PIWIK.)

@1stone commented on August 7th 2015

For reference, this is the original request sent from log-importer-py to PIWIK (URL was obfuscated):

{
  "requests": [{
    "apiv": "1",
    "url": "http://www.example.com/.../some/where",
    "cvar": "{\"1\": [\"HTTP-code\", \"200\"]}",
    "cdt": "2015-07-14 02:27:53",
    "cip": "66.249.64.238",
    "urlref": "",
    "bw_bytes": 16541,
    "rec": "1",
    "_cvar": "{\"1\": [\"Bot\", \"Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"]}",
    "idsite": "18",
    "bots": "1",
    "ua": "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "dp": "0"
  }, {
    "apiv": "1",
    "url": "http://www.example.com/.../some/where",
    "cvar": "{\"1\": [\"HTTP-code\", \"200\"]}",
    "cdt": "2015-07-14 02:27:56",
    "cip": "66.249.64.238",
    "urlref": "",
    "bw_bytes": 19456,
    "rec": "1",
    "_cvar": "{\"1\": [\"Bot\", \"Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"]}",
    "idsite": "18",
    "bots": "1",
    "ua": "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "dp": "0"
  }],
  "token_auth": "1ea06bef5839065d1159d8f5a941d40a"
}
@tsteur commented on August 7th 2015 Owner

I just mentioned to change the password and to generate a new token as anyone could otherwise log into your Piwik with this token.

Please use something like this to reproduce the error:

I did and it works. I presume it is related to LoginLdap maybe. @diosmosis do you have LoginLdap setup? If so do you mind having a quick look and trying to do a bulk request with token_auth and cdt/cip?

@diosmosis commented on August 7th 2015 Member

I don't have loginldap setup (also currently eating), but LoginLdap should be at 3.0+ so 1.3.4 is very old. Assuming everyone's talking about the same plugin though, 1.3.4 should not actually work with the latest piwik.

@1stone commented on August 7th 2015

The posted token is a fake, so no problem there. In my tests I replaced this with the real token, of course.

The cause of the problem is obviously the LoginLdap plugin!
I deactivated it, and the service immediately responded properly.

So, either there is a bug in the LoginLdap plugin, not honoring the locally stored auth tokens, or it is a misconfiguration.
However, from looking at the plugin configuration, I don't recognize any related option that may affect this.

@tsteur commented on August 7th 2015 Owner

Can you try to update the plugin?

@1stone commented on August 7th 2015

Sorry, that was a typo.
LoginLdap is on 3.1.4.

@tsteur commented on August 7th 2015 Owner

ok, so it sounds like a bug. Thx!

@diosmosis commented on August 7th 2015 Member

@1stone can you post your LoginLdap configuration (a screenshot of the settings page will do)?

@1stone commented on August 7th 2015

ldap_settings

@diosmosis commented on August 7th 2015 Member

@1stone Thanks, I will try to reproduce and fix the issue over the weekend. Can you tell me if the token auths you tried to track with belonged to LDAP users or users only in the Piwik DB?

@1stone commented on August 7th 2015

The token auths I tried belonged to local as well as LDAP users (all with superuser_access=1).
All showed the same behavior.

@diosmosis commented on August 9th 2015 Member

@1stone I believe I fixed the issue in the LoginLdap plugin. I will release a new version after looking at some other issues, in the meantime the fix for your specific problem is here: https://github.com/piwik/plugin-LoginLdap/commit/360671c46893410223264e32a1d5de4feb62f3eb

@1stone commented on August 10th 2015

I applied the suggested changes and can confirm that it fixes the problem.

Thanks for your support!

@diosmosis commented on August 25th 2015 Member

Fixed in version 3.1.5 of LoginLdap.

This Issue was closed on August 25th 2015
Powered by GitHub Issue Mirror