Sir Iain Livingstone made the comment during a meeting of the Scottish Police Authority (SPA) board on Thursday, saying the first step to change is acknowledging the issue.
First Minister Humza Yousaf described his admission as “monumental” and “historic”.
Sir Iain was keen to stress he was not pointing the finger at individual officers, but conceded some communities in Scotland are not getting the service from police that they should be.
With just months left before he retires from the force, Sir Iain has come in for criticism over the timing of his statement, including from former SPA board member Moi Ali.
Asked on the BBC’s Good Morning Scotland radio programme on Friday if the announcement could have come sooner, the chief constable said: “Could I have done this earlier? Could I have myself got to this position of accepting and recognising institutional discrimination?
“Well perhaps, perhaps I could have, and I accept that.
“But what I do say is that I’ve always been committed to driving equality, diversity and inclusion – we’ve got far greater representation now in policing than we’ve ever had.
“Lots more to do, but I think I leave the organisation in a far better place than I found it.”
Sir Iain insisted his statement does not represent a failure of his leadership, adding: “I will leave office later in August, after having been chief constable for almost six years, I think we’ve got Police Scotland into a far more stable place.”
He said that of more than 570 murders since the creation of Police Scotland 10 years ago, just one remains undetected, describing the record of the service as “massively impressive” and paying tribute to community officers, campus officers and detectives.
But Callum Steele, the former head of the Scottish Police Federation (SPF) – which represents rank-and-file officers – was excoriating in his criticism of Sir Iain’s admission, accusing him of “extreme sabotage” that will “devastate the gossamer-thin remnants of any morale in the police service”.
Sir Iain conceded his statement would have be “difficult to hear”.
He added: “It’s difficult to say.
“As I said, I’m a police officer myself for almost 31 years and my loyalty to my colleagues and officers and staff who I’ve worked with for many, many years, people who I know, I know their qualities, I know their values, and it’s difficult to hear.
“But you need to be clear on what I’m actually saying. I am not condemning officers and staff, I’m actually, if anything, looking at the organisation for which I’m responsible and it’s actually taken me time to have that acceptance and that realisation for the reasons I’ve said.”
Sir Iain stressed he is not “condemning colleagues” or saying that “no progress has been made since the 90s”.
Meanwhile, the current chairman of the SPF said it will be harder for officers this weekend following Sir Iain’s comments.
David Threadgold told Good Morning Scotland: “The vast majority of the police officers who heard the words from the chief constable yesterday, and by extension members of the public, their perception is that the chief constable was identifying and labelling them as institutionally racist.
“That is simply not the case.
“The distinction, the nuance the chief constable is trying to make between organisational issues and individual collective responsibility was missed in the delivery.
“That is a really important distinction to make because I believe that the role of police officers now in the communities will have been made more difficult by the comments of the chief constable.”
Mr Threadgold also said while only a minority of people have had issues with officers, those people should not be ignored.