Active Directory Users Sync

edited January 2016 in General
Hello,

I've been trying to sync some users from Active Directory using the command "multiotp.php -debug -display-log -ldap-users-sync" but I always get this error:
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 130968 bytes) in /etc/multiotp/multiotp.php on line 56

I'm guessing it has to do with caching the users groups; I tried increasing the memory for php in "/etc/php.ini" but that didn't help, I also tried setting "_real_primarygroup" to false in "multiotp.class.php" but that didn't help either.

I'm wondering if anyone has a way to get multiOTP to work with a large AD.

Here are the related configurations for reference:
ldap_account_suffix=
ldap_activated=1
ldap_base_dn=DC=mytestdomain,DC=com
ldap_bind_dn=CN=svc,OU=Accounts,OU=Management,DC=mytestdomain,DC=com
ldap_cn_identifier=sAMAccountName
ldap_domain_controllers=10.10.10.10
ldap_group_attribute=memberOf
ldap_group_cn_identifier=sAMAccountName
ldap_hash_cache_time=604800
ldap_in_group="VPNUsers"
ldap_network_timeout=10
ldap_port=389
ldap_server_password=password1
ldap_server_type=1
ldap_ssl=0
ldap_time_limit=30

Thanks.

Comments

  • Hello,
    What is the size of your large AD directory (how many users and how many groups) ?
    The last beta version is much more optimized for larger AD directory, you can download it here: http://download.multiotp.net/beta/
    Have a try, and in any case, thanks to keep us in touch concerning the size of your AD.
    Best regards,
    Andre
  • The AD directory has about 16000 entries, and I can confirm that the beta can handle it :)

    However, I'm now having a strange problem when syncing some groups; I've noticed that users that aren't in the group get synced, while other groups don't sync at all!
    I'm wondering if this issue is caused by whitespaces and/or special characters (hyphens?) in the group names, since groups that don't have spaces in their names seem to sync just fine.
  • Hello,
    this will be corrected in the next release. It will be availlable to download on the 4th of April.

    Best regards,
    Yann
  • edited September 2016
    Hi, same problem, huge AD, trying to take only 1 group (~50 users) (stable and beta)

    >multiotp -debug -display-log -ldap-users-sync

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate
    35 bytes) in D :\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 16954

    One more thing - there's a logo - "open source" on top of this page, is there any way one can get the source code? 
  • Hello, we will make some tests and get back to you. Source code is availlable here : https://www.multiotp.net/?website=multiotp&rl=2&ll=&it=101637&language=en

    Have a nice day
  • edited November 2016
    Hi,
    still waiting for yout response : D
    Maybe you can at least compile multiotp version with the memory limit of 4-8gb instead of 512mb? (Hope it'll solve the issue).
    We can discuss the donation amount for the product development : )
  • edited November 2016
    I still experience the same problem, even if there is only one user in group.
    Guys, i really need your fast help, cause the decision of using motp or not will be made in couple of days. I really like it, especially for rdp purposes, but i need to make it work.

    c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-18 12:33:09 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19123 

    c:\motp>multiotp -version 
    multiOTP 5.0.2.6 (2016-11-04)
  • Hello Rager,

    Can you give us a short description of your AD, especially:
    - how many users in total
    - how many groups in total
    - how many users in a group (for the biggest group)

    Best regards,

    Andre
  • Hello,

    The issue is based on the algorithm to find which users are in which groups. The algorithm is done with cache in order to have high efficiency on small/medium AD, but when there is a lot of groups, the cache is going crazy. The dfficulty is that we are also handling groups in groups, which can takes a lot of time.

    Synchronizing any amount of users is not a problem, it takes about 1 min / 1000 users (about 100 minutes for 100'000 users), and the used memory don't grow.

    We are trying another adaptative algorithm for a big amount groups, but we still need to be compatible with both Linux and Windows, and they are not handling the groups the same manner.

    Best regards,

    Andre
  • edited November 2016
    Hello, our AD current security groups:

    PS C:\Users\rager> echo (GET-ADGroup -filter 'GroupCategory -eq "security"').count

    202

    PS C:\Users\rager> get-aduser -filter * | measure-object | select-object count

    Count

    -----

      523

    Also we have 24 custom OU's, that host those 202 groups.

    I'd really appreciate any kind of solution, so that i can demonstrate it working on production in the morning of Wednesday, so that big boss'll allow to use it and i can demand a donation for your project.


    May be non-cache version, or huge memory consumption version (eg 128gb :D). I just need to sync it once for production AD 2fa presentation, cause it is really good for RDP on hyper-v, especially when we need to use 2fa only for rdp connection via rd gateway and non 2fa for rdp connection from lan.
  • We are doing some tests on the new algorithm, as huge memory consumption is not a long term solution (and not a long term too :-). Stay tuned, we come back soon to you.
  • That sounds great! ^^
  • New 5.0.3.2-beta-1 will be available in about two hours... :-)
  • c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:2083 Multiotp::WriteConfigData() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:65342 Multiotp::UpgradeSchemaIfNeeded() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData main:1 include() 

    LOG 2016-11-23 04:15:18 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19716 

    c:\motp>multiotp -version 

    LOG 2016-11-23 04:19:20 debug Debug Debug: *parameter(s) received: -version 
    multiOTP 5.0.3.1-beta-2 (2016-11-16) 
    19 *INFO: Requested operation successfully done
  • Hello Rager,
    Have a look here : http://download.multiotp.net/beta/
    5.0.3.2-beta-1 is available
    Regards,
    Andre
  • c:\motp>multiotp -debug -display-log -ldap-users-sync 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:2098 Multiotp::WriteConfigData() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData D:\Data\projects\mul 
    tiotp\phc-cli\multiotp.windows.php:65388 Multiotp::UpgradeSchemaIfNeeded() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData main:1 include() 

    LOG 2016-11-23 07:06:36 debug Debug Debug: *parameter(s) received: -debug -displ 
    ay-log -ldap-users-sync 

    Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
    261904 bytes) in D:\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
    19766 



  • c:\motp>multiotp -version 

    LOG 2016-11-23 07:16:38 debug Debug Debug: *parameter(s) received: -version 
    multiOTP 5.0.3.2-beta-1 (2016-11-22) 
    19 *INFO: Requested operation successfully done 
  • Hello,
    In debug mode, the first information are expected (it's regular debug information).
    The exhausted memory is due to recursive groups detection.
    I have done a new beta build with the following limitations:
    - the primary group of a user cannot be used as a filtering group for multiOTP
    - the users must be attributed directly to the filtering group(s), and not in a group that contains recursively filtering group(s)
    The new beta is multiOTP 5.0.3.2-beta-2.
    http://download.multiotp.net/beta/
    Regards,
  • edited November 2016
    hooray! finally it works!
    there are some issues, but i can sync users on production ad now! thank you so much, i'll contact you soon
  • Hello,
    Cool, can you please give us the issues ?
    Regards,
    Andre
This discussion has been closed.