If you want to subscribe to this forum, use your Facebook account, or send us an email to forum - at - multiotp - dot - net and we will send you back an invitation.

Active Directory Users Sync

edited January 2016 in General
Hello,

I've been trying to sync some users from Active Directory using the command "multiotp.php -debug -display-log -ldap-users-sync" but I always get this error:
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 130968 bytes) in /etc/multiotp/multiotp.php on line 56

I'm guessing it has to do with caching the users groups; I tried increasing the memory for php in "/etc/php.ini" but that didn't help, I also tried setting "_real_primarygroup" to false in "multiotp.class.php" but that didn't help either.

I'm wondering if anyone has a way to get multiOTP to work with a large AD.

Here are the related configurations for reference:
ldap_account_suffix=
ldap_activated=1
ldap_base_dn=DC=mytestdomain,DC=com
ldap_bind_dn=CN=svc,OU=Accounts,OU=Management,DC=mytestdomain,DC=com
ldap_cn_identifier=sAMAccountName
ldap_domain_controllers=10.10.10.10
ldap_group_attribute=memberOf
ldap_group_cn_identifier=sAMAccountName
ldap_hash_cache_time=604800
ldap_in_group="VPNUsers"
ldap_network_timeout=10
ldap_port=389
ldap_server_password=password1
ldap_server_type=1
ldap_ssl=0
ldap_time_limit=30

Thanks.

Comments

  • Hello,
    What is the size of your large AD directory (how many users and how many groups) ?
    The last beta version is much more optimized for larger AD directory, you can download it here: http://download.multiotp.net/beta/
    Have a try, and in any case, thanks to keep us in touch concerning the size of your AD.
    Best regards,
    Andre
  • The AD directory has about 16000 entries, and I can confirm that the beta can handle it :)

    However, I'm now having a strange problem when syncing some groups; I've noticed that users that aren't in the group get synced, while other groups don't sync at all!
    I'm wondering if this issue is caused by whitespaces and/or special characters (hyphens?) in the group names, since groups that don't have spaces in their names seem to sync just fine.
  • Hello,
    this will be corrected in the next release. It will be availlable to download on the 4th of April.

    Best regards,
    Yann
  • edited September 2016
    Hi, same problem, huge AD, trying to take only 1 group (~50 users) (stable and beta)

    >multiotp -debug -display-log -ldap-users-sync

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate
    35 bytes) in D :\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 16954

    One more thing - there's a logo - "open source" on top of this page, is there any way one can get the source code? 
  • Hello, we will make some tests and get back to you. Source code is availlable here : https://www.multiotp.net/?website=multiotp&rl=2&ll=&it=101637&language=en

    Have a nice day
  • edited November 2016
    Hi,
    still waiting for yout response : D
    Maybe you can at least compile multiotp version with the memory limit of 4-8gb instead of 512mb? (Hope it'll solve the issue).
    We can discuss the donation amount for the product development : )
  • edited November 2016
    I still experience the same problem, even if there is only one user in group.
    Guys, i really need your fast help, cause the decision of using motp or not will be made in couple of days. I really like it, especially for rdp purposes, but i need to make it work.

    c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-18 12:33:09 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19123 

    c:\motp>multiotp -version 
    multiOTP 5.0.2.6 (2016-11-04)
  • Hello Rager,

    Can you give us a short description of your AD, especially:
    - how many users in total
    - how many groups in total
    - how many users in a group (for the biggest group)

    Best regards,

    Andre
  • Hello,

    The issue is based on the algorithm to find which users are in which groups. The algorithm is done with cache in order to have high efficiency on small/medium AD, but when there is a lot of groups, the cache is going crazy. The dfficulty is that we are also handling groups in groups, which can takes a lot of time.

    Synchronizing any amount of users is not a problem, it takes about 1 min / 1000 users (about 100 minutes for 100'000 users), and the used memory don't grow.

    We are trying another adaptative algorithm for a big amount groups, but we still need to be compatible with both Linux and Windows, and they are not handling the groups the same manner.

    Best regards,

    Andre
  • edited November 2016
    Hello, our AD current security groups:
    PS C:\Users\rager> echo (GET-ADGroup -filter 'GroupCategory -eq "security"').count
    202
    PS C:\Users\rager> get-aduser -filter * | measure-object | select-object count
    Count
    -----
      523
    Also we have 24 custom OU's, that host those 202 groups.
    I'd really appreciate any kind of solution, so that i can demonstrate it working on production in the morning of Wednesday, so that big boss'll allow to use it and i can demand a donation for your project.

    May be non-cache version, or huge memory consumption version (eg 128gb :D). I just need to sync it once for production AD 2fa presentation, cause it is really good for RDP on hyper-v, especially when we need to use 2fa only for rdp connection via rd gateway and non 2fa for rdp connection from lan.
  • We are doing some tests on the new algorithm, as huge memory consumption is not a long term solution (and not a long term too :-). Stay tuned, we come back soon to you.
  • That sounds great! ^^
  • New 5.0.3.2-beta-1 will be available in about two hours... :-)
  • c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:2083 Multiotp::WriteConfigData() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:65342 Multiotp::UpgradeSchemaIfNeeded() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData main:1 include() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19716 

    c:\motp>multiotp -version 

    LOG 2016-11-23 04:19:20 debug Debug Debug: *parameter(s) received: -version 
    multiOTP 5.0.3.1-beta-2 (2016-11-16) 
    19 *INFO: Requested operation successfully done
  • Hello Rager,
    Have a look here : http://download.multiotp.net/beta/
    5.0.3.2-beta-1 is available
    Regards,
    Andre
  • c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:2098 Multiotp::WriteConfigData() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:65388 Multiotp::UpgradeSchemaIfNeeded() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData main:1 include() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19766 



  • c:\motp>multiotp -version 

    LOG 2016-11-23 07:16:38 debug Debug Debug: *parameter(s) received: -version 
    multiOTP 5.0.3.2-beta-1 (2016-11-22) 
    19 *INFO: Requested operation successfully done 
  • Hello,
    In debug mode, the first information are expected (it's regular debug information).
    The exhausted memory is due to recursive groups detection.
    I have done a new beta build with the following limitations:
    - the primary group of a user cannot be used as a filtering group for multiOTP
    - the users must be attributed directly to the filtering group(s), and not in a group that contains recursively filtering group(s)
    The new beta is multiOTP 5.0.3.2-beta-2.
    http://download.multiotp.net/beta/
    Regards,
  • edited November 2016
    hooray! finally it works!
    there are some issues, but i can sync users on production ad now! thank you so much, i'll contact you soon
  • Hello,
    Cool, can you please give us the issues ?
    Regards,
    Andre
This discussion has been closed.