Active Directory Users Sync

Harvii

Hello,

I've been trying to sync some users from Active Directory using the command "multiotp.php -debug -display-log -ldap-users-sync" but I always get this error:
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 130968 bytes) in /etc/multiotp/multiotp.php on line 56

I'm guessing it has to do with caching the users groups; I tried increasing the memory for php in "/etc/php.ini" but that didn't help, I also tried setting "_real_primarygroup" to false in "multiotp.class.php" but that didn't help either.

I'm wondering if anyone has a way to get multiOTP to work with a large AD.

Here are the related configurations for reference:
ldap_account_suffix=
ldap_activated=1
ldap_base_dn=DC=mytestdomain,DC=com
ldap_bind_dn=CN=svc,OU=Accounts,OU=Management,DC=mytestdomain,DC=com
ldap_cn_identifier=sAMAccountName
ldap_domain_controllers=10.10.10.10
ldap_group_attribute=memberOf
ldap_group_cn_identifier=sAMAccountName
ldap_hash_cache_time=604800
ldap_in_group="VPNUsers"
ldap_network_timeout=10
ldap_port=389
ldap_server_password=password1
ldap_server_type=1
ldap_ssl=0
ldap_time_limit=30

Thanks.

AndreL

Hello,
What is the size of your large AD directory (how many users and how many groups) ?
The last beta version is much more optimized for larger AD directory, you can download it here: http://download.multiotp.net/beta/
Have a try, and in any case, thanks to keep us in touch concerning the size of your AD.
Best regards,
Andre

Harvii

The AD directory has about 16000 entries, and I can confirm that the beta can handle it

However, I'm now having a strange problem when syncing some groups; I've noticed that users that aren't in the group get synced, while other groups don't sync at all!
I'm wondering if this issue is caused by whitespaces and/or special characters (hyphens?) in the group names, since groups that don't have spaces in their names seem to sync just fine.

YannJ

Hello,
this will be corrected in the next release. It will be availlable to download on the 4th of April.

Best regards,
Yann

Rager

Hi, same problem, huge AD, trying to take only 1 group (~50 users) (stable and beta)

>multiotp -debug -display-log -ldap-users-sync

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate
35 bytes) in D :\Data\projects\multiotp\phc-cli\multiotp.windows.php on line 16954

One more thing - there's a logo - "open source" on top of this page, is there any way one can get the source code? 

YannJ

Hello, we will make some tests and get back to you. Source code is availlable here : https://www.multiotp.net/?website=multiotp&rl=2&ll=&it=101637&language=en

Have a nice day

Rager

Hi,

still waiting for yout response : D

Maybe you can at least compile multiotp version with the memory limit of 4-8gb instead of 512mb? (Hope it'll solve the issue).

We can discuss the donation amount for the product development : )

Rager

I still experience the same problem, even if there is only one user in group.

Guys, i really need your fast help, cause the decision of using motp or not will be made in couple of days. I really like it, especially for rdp purposes, but i need to make it work.

c:\motp>multiotp -debug -display-log -ldap-users-sync 

LOG 2016-11-18 12:33:09 debug Debug Debug: *parameter(s) received: -debug -displ 
ay-log -ldap-users-sync 

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
261904 bytes) in \Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
19123 

c:\motp>multiotp -version 
multiOTP 5.0.2.6 (2016-11-04)

adminf

Hello Rager,

Can you give us a short description of your AD, especially:
- how many users in total
- how many groups in total
- how many users in a group (for the biggest group)

Best regards,

Andre

adminf

Hello,

The issue is based on the algorithm to find which users are in which groups. The algorithm is done with cache in order to have high efficiency on small/medium AD, but when there is a lot of groups, the cache is going crazy. The dfficulty is that we are also handling groups in groups, which can takes a lot of time.

Synchronizing any amount of users is not a problem, it takes about 1 min / 1000 users (about 100 minutes for 100'000 users), and the used memory don't grow.

We are trying another adaptative algorithm for a big amount groups, but we still need to be compatible with both Linux and Windows, and they are not handling the groups the same manner.

Best regards,

Andre

Rager

Hello, our AD current security groups:

PS C:\Users\rager> echo (GET-ADGroup -filter 'GroupCategory -eq "security"').count

202

PS C:\Users\rager> get-aduser -filter * | measure-object | select-object count

Count

-----

  523

Also we have 24 custom OU's, that host those 202 groups.

I'd really appreciate any kind of solution, so that i can demonstrate it working on production in the morning of Wednesday, so that big boss'll allow to use it and i can demand a donation for your project.

May be non-cache version, or huge memory consumption version (eg 128gb ). I just need to sync it once for production AD 2fa presentation, cause it is really good for RDP on hyper-v, especially when we need to use 2fa only for rdp connection via rd gateway and non 2fa for rdp connection from lan.

adminf

We are doing some tests on the new algorithm, as huge memory consumption is not a long term solution (and not a long term too :-). Stay tuned, we come back soon to you.

Rager

That sounds great! ^^

Rager

Is there any news?

Rager

No luck?

adminf

New 5.0.3.2-beta-1 will be available in about two hours... :-)

Rager

c:\motp>multiotp -debug -display-log -ldap-users-sync 

LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData \Data\projects\mul 
tiotp\phc-cli\multiotp.windows.php:2083 Multiotp::WriteConfigData() 

LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData \Data\projects\mul 
tiotp\phc-cli\multiotp.windows.php:65342 Multiotp::UpgradeSchemaIfNeeded() 

LOG 2016-11-23 04:15:18 debug Debug Debug: *WriteConfigData main:1 include() 

LOG 2016-11-23 04:15:18 debug Debug Debug: *parameter(s) received: -debug -displ 
ay-log -ldap-users-sync 

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
261904 bytes) in \Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
19716 

c:\motp>multiotp -version 

LOG 2016-11-23 04:19:20 debug Debug Debug: *parameter(s) received: -version 
multiOTP 5.0.3.1-beta-2 (2016-11-16) 
19 *INFO: Requested operation successfully done

adminf

Hello Rager,
Have a look here : http://download.multiotp.net/beta/
5.0.3.2-beta-1 is available
Regards,
Andre

Rager

c:\motp>multiotp -debug -display-log -ldap-users-sync 

LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData \Data\projects\mul 
tiotp\phc-cli\multiotp.windows.php:2098 Multiotp::WriteConfigData() 

LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData \Data\projects\mul 
tiotp\phc-cli\multiotp.windows.php:65388 Multiotp::UpgradeSchemaIfNeeded() 

LOG 2016-11-23 07:06:36 debug Debug Debug: *WriteConfigData main:1 include() 

LOG 2016-11-23 07:06:36 debug Debug Debug: *parameter(s) received: -debug -displ 
ay-log -ldap-users-sync 

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 
261904 bytes) in \Data\projects\multiotp\phc-cli\multiotp.windows.php on line 
19766 

Rager

c:\motp>multiotp -version 

LOG 2016-11-23 07:16:38 debug Debug Debug: *parameter(s) received: -version 
multiOTP 5.0.3.2-beta-1 (2016-11-22) 
19 *INFO: Requested operation successfully done 

adminf

Hello,
In debug mode, the first information are expected (it's regular debug information).
The exhausted memory is due to recursive groups detection.
I have done a new beta build with the following limitations:
- the primary group of a user cannot be used as a filtering group for multiOTP
- the users must be attributed directly to the filtering group(s), and not in a group that contains recursively filtering group(s)
The new beta is multiOTP 5.0.3.2-beta-2.
http://download.multiotp.net/beta/
Regards,

Rager

hooray! finally it works!

there are some issues, but i can sync users on production ad now! thank you so much, i'll contact you soon

adminf

Hello,
Cool, can you please give us the issues ?
Regards,
Andre